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This  guidebook  was  prepared  as  part  of  the  Software  Acquisition  Engineering 
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It  reflects  the  latest  DOD  published  policy  on  software  quality  assurance  while 
offering  a  practical  and  cost  effective  approach  to  total  life  cycle  software 
quality  assurance. 

This  guidebook  is  one  of  a  series  intended  to  assist  the  Air  Force  Program  Office 
and  engineering  personnel  in  software  acquisition  engineering  for  automatic  test 
equipment  and  training  simulators.  Titles  of  other  guidebooks  in  the  series  are 
listed  in  the  introduction.  These  guidebooks  will  be  revised  periodically  to 
reflect  changes  in  software  acquisition  policies  and  feedback  from  users. 
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subsequent  changes  to  the  command  media  may  invalidate  such  interpretations  the 
reader  should  also  consult  applicable  government  documents  representing  authorized 
software  acquisition  engineering  processes. 

This  guidebook  contains  alternative  recommendations  concerning  methods  for 
cost-effective  software  acquisition.  The  intent  is  that  the  reader  determine  the 
degree  of  appl icabi 1 ity  of  any  alternative  based  on  specific  requirements  of  the 
software  acquisition  with  which  he  is  concerned.  Hence  the  guidebook  should  only  be 
implemented  as  advisory  rather  than  as  mandatory  or  directive  in  nature. 
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Section  1.0  INTRODUCTION 


AF  Policy  (AF-74-1)  defines  Quality 
Assurance  (QA)  as  a  discipline  which  pro¬ 
vides  adequate  assurance  that  material, 
+  ,suPPl i es  and  services  conform  to 

established  technical  requirements  and 
achieve  satisfactory  results."  This 
definition  transcends  earlier  interpreta¬ 
tions  which  assumed  that  QA  should 
merely  verify  conformance  to  specifica¬ 
tions.  Software  Quality  Assurance  (SWQA) 
concepts  today,  which  are  truly  cost 
effective  in  the  long  term,  reflect  a 
total  life  cycle  approach.  Accordingly, 
methoos  presented  in  this  guidebook  are 
designed  to  achieve  not  only  quality  of 
conformance,  but  quality  of  specifica¬ 
tion  and  quality  of  design  as  well.  True 
quality  software  satisfies  a  customer's 
needs  not  simply  a  specification.  SWQA 
therefore,  is  an  organizational ly  inter¬ 
dependent  set  of  documented  controls 
spanning  all  phases  of  the  software  life 
cycle.  These  controls  are  designed  to 
minimize  the  quantity  of  post-delivery 
software  deficiencies  traditionally  so 
prevalent  in  software  systems.  SWQA  is 
not  limited  to  the  activities  of  a 
single  contractor  functional  organiza- 
tion  nor  to  activities  associated 

pIthCt,y  c.,™th.  Pr°9ram  acceptance. 
Rather,  SWQA  is  a  mul t iorganizational 
effort  administered  under  the  auspices 
of  QA  which  spans  all  phases  of  system 
acquisition,  and  covers  all  activities 
which  assures  development  and  delivery 
of  high  quality  software.  Qualitv  soft¬ 
ware  is  characterized  by: 


a.  Complete,  unambiguous 
ments  specifications. 

b.  Readily  verifiable  design. 


requi  re- 


c.  Ease  of  program  change  and 
maintenance. 

d.  Documentation  compliant  with 
programming  conventions  and  standards. 


e.  Accurate  and  complete  records  of 
requirements  satisfaction. 

f.  Minimal  post-delivery  errors. 

The  Department  of  Defense,  as  reflected 
in  numerous  publications  (see  Section 
cun*  nas  rec°9nized  the  necessity  of 
SWQA,  as  vital  to  the  on-time, 
wi thin-target  cost  delivery  of  hiqh 
quality  software.  This  guidebook 
sents  methods  of  implementation 
pliant  with  that  policy. 


pre- 

com- 


1.1  PURPOSE 

Jt.  .Vs  ,the  specific  purpose  of  this 
guidebook  to  present  SWQA  methodology 

ffllr?red^  t°  .Automatic  Test  Equipment 
(Ait)  and  Training  Simulator  (TS)  system 
acquisitions.  The  guidebook  will  be  used 
by  AF  acquisition  engineering  personnel 

cHn«COnCeive’  tailor>  specify  and  enforce 
SWQA  requirements  in  Request  for  Pro¬ 
posal  (RFP's)  and  other  contractual  docu¬ 
ments  and  specifications.  It  is  also  use¬ 
ful  in  preparing  checklists  for  proposal 
evaluations  and  contractor  surveillance 
activities.  It  should  be  recognized  that 
acquiring  ATE  or  TS  software  is  an  inte¬ 
gral  part  of  a  larger  system  acquisi¬ 
tion.  Accordingly,  the  guidebook  con- 
all  system  requirements  impacting 
SWQA  including  applicable  hardware  devel¬ 
opment  requirements,  configuration  man¬ 
agement,  data  and  design  reviews, 
audits,  and  all  modes  of  requirements 
satisfaction.  The  ultimate  objective  of 
these  controls  is  to  enforce  disciplined 
/rn£r\er  Pro9>"am  Configuration  Item 
(CPCI)  development  controls  and  to  pro¬ 
vide  official  objective  evidence  of  CPCI 
requirement  satisfaction. 

1.2  SCOPE 

Icnri  Software  Acquisition  Engineering 
(SAE)  guidebook  is  one  in  the  guidebook 
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f?canSvrATrtedHfc?c9r0und  systems»  speci- 
are  as  follow??  9Uldebook  ^tles 

Software  Cost  Measuring  and  Reporting 
Requirements  Specif ication  9 

Contracting  for  Software  Acquisition 
Statement  of  Work  (SOW)  and  Requests 
for  Proposal  (RFP)  9 

Regulations,  Specifications  and 
standards 

Measuring  and  Reporting  Software 
status 

Computer  Program  Documentation 
Requirements 

Software  Quality  Assurance 
verification 

Validation  and  Certification 
Computer  Program  Maintenance 
Software  Configuration  Management 
Reviews  and  Audits 

Management  Reporting  by  the  Software 
Director 

The  SWQA  methods  discussed  in  *k< 
guidebook  encompass  the  following  areas;5 

effort ESt1mat1n9  and  planm'n9  the  SWQA 


and'coSpl5 l a'Jce  stalls f  f  aM  1 1 ty 

account  ing1^Uratl°r'  Ven"f  and 


Verification 

testing 


and 


val idation 


e-  Media  identification  and  control 

test  "tools  S°ftWare  developmental  and 

These  controls  implement  AF  reaul it innc 
and  specifications'  relating  to  ^nnuter 
program  development  and  quality  S 

into  fJJnCt\°n  15  documented,  integrated 
nt?  formal  acquisition  planning  and 
implemented  throughout  all  ATF  and  tc 

ifsst  ir  <*«« • 

tten  for  managers,  engineers  and 


S?  iF<™> ■skm  & 

acceptancefVrdroperati‘ofial0deploymertfnt’ 

1*3  TS  AND  ATE  OVERVIEW 

TS15  aSnedCtl0ATEPr°VideS  3  bn'ef  sk*tch  of 

including  ffi 

programs  associated  with  each.  P 

1.3.1  Trainer  Simulator  System 
Characteristics 


specialized73^  a  ’S  3  con,bl' nation  of 

and  improve  the  tprhm*  *  develop 

rj“  ***'r  31 

i  -  ii 

digital  processing  capability.  H  y 

The  computation  system,  integral  tn  *ho 
crew  training  simulator,  consists  of 

c°ormpu°t  ?ng 

provide  efflSenT  ese  of  1  t0 

£*  ,H0L> • 

ES:?'r“= 

nshed 

sta  inn  T  thG  1nstructor  operator 
station;  performs  a  real-tine  <ni„e 
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Figure  1.3-1.  Typical  Crew  Training  Simulator 
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criteria.  Since  training  simulators  are 
a  combination  of  i nterdependent  hardware 
and  software,  a  joint  development  effort 
is  required.  As  the  complexity  of  train¬ 
ing  simulators  increases,  simulation 
software  continues  to  grow  in  complex¬ 
ity,  size,  and  cost.  Software  costs  can 
and  do  exceed  computer  hardware  costr  in 
many  cases.  Therefore,  it  is  impere  ive 
that  the  simulation  software  acquis'  ion 
engineering  process  be  subjected  to 
formal  system  engineering  planning  and 
discipline  to  ensure  effective  and 
efficient  simulator  procurement. 

1.3.2  Automatic  Test  Equipment 
Characteristics 

ATE  consists  of  electronic  devices  cap¬ 
able  of  automatically  or  semi- 
automatical  ly  generating  and  indepen¬ 
dently  furnishing  programmed  stimuli, 
measuring  selected  parameters  of  an  item 
being  tested  and  making  a  comparison  to 
accept  or  reject  the  measured  values  in 
accordance  with  predetermined  limits. 
ATE  is  used  in  place  of  manual  devices, 
either  because  it  is  more  cost  effec¬ 
tive,  or  the  item  being  tested  requires 
the  speed  and  timing  which  only  an  auto¬ 
matic  tester  can  achieve. 

Figure  1.3-2  shews  the  typical  com¬ 
ponents  of  an  ATE  system.  Note  that 
there  are  both  hardware  and  software 
elements  involved.  Most  of  the  elements 
shown  will  be  found  in  one  form  or 
another  in  the  majority  of  ATE  systems. 
The  controls  and  displays  section  con¬ 
sists  of  the  computer  and  peripheral 
devices  like  control  panels,  magnetic 
tape  cassettes  or  disks,  a  CRT  and  key¬ 
board,  and  usually  a  small  printer.  The 
computer,  as  controlled  by  software,  per¬ 
forms  the  tasks  of  operating  the  peri¬ 
pheral  devices,  switching  test  stimuli 
on  and  off,  and  measuring  and  comparing 
responses  of  the  unit  under  test  (UUT) 
to  predetermined  values.  The  operator 
will  maintain  ultimate  control  of  the 
testing  process  through  some  of  the 
peripherals;  however,  his  interaction  is 
usually  minimal  since,  by  definition, 
the  automatic  test  feature  was  selected 


in  preference  to  an  operator-controlled 
test  system.  The  interface  equipment  may 
be  a  combination  of  hardware  and  soft¬ 
ware.  It  is  normally  designed  to  allcw  a 
single  configuration  of  ATE  to  be  used 
for  testing  several  articles  of  system 
equipment.  The  maintenance  level  being 
supported  by  the  ATE  is  determined  by 
logistic  planning. 

The  importance  of  the  software  portion 
of  the  ATE  system  should  not  be  mini¬ 
mized  since  both  the  application  of  the 
test  stimuli  and  the  measurement  of  the 
result  are  achieved  via  software.  In 
some  cases  (not  always),  arbitrary  func¬ 
tion  generation  and  complicated  wave 
analysis  is  also  accomplished  by 
software. 

1.4  GUIDEBOOK  ORGANIZATION 

Because  SV/QA  is  a  multi-phased,  multi- 
organizational  effort,  the  guidebook  is 
organized  to  follow  a  logical  software 
development  flow  within  a  typical  ATE  or 
TS  system  acquisition  process.  It  is 
noted  that  not  all  ATE  and  TS  system 
acquisitions  are  typical.  Some  may  have 
fewer  formal  phases  of  development  than 
others.  Some  are  separate  contracts. 
Others  are  contracted  for  as  changes  to 
prime  weapon  system  contracts.  Some 
acquisitions  may  have  complex  con¬ 
tracting  arrangements  involving  multiple 
associate  contractors.  All  such  acquisi¬ 
tions  however,  follow  some  variation  of 
the  basic  phases  (Analysis,  Design,  Code 
and  Checkout,  Test  and  Integration, 
Installation,  Operation  and  Support)  in 
their  development. 

In  discussing  SWQA  methodology  in 
sequential  fashion,  considerations  are 
presented  unique  to  AiE  and  TS  software 
development.  The  i nter-rel ati onship  of 
PO  and  contractor  responsibilities  is 
discussed  in  Section  3.0;  followed  by 
treatment  of  software  QA  tasks  as  they 
are  encountered  in  a  logical  software 
development  sequence,  in  Sections  4.0 
through  6.0.  Sections  7.0  through  II. 0 
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include  a  bibliography,  topic  vs.  Govern¬ 
ment  document  matrix,  glossary,  abbrevia¬ 
tions  and  acronyms  list,  and  a  subject 
index  respectively. 
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Section  2.0  APPLICABLE  DOCUMENTS 


The  following  is  a  list  of  government 
and  industry  publications  relevant  to 
software  quality  assurance,  ATE  and  TS. 

AFR  800-14,  Acquisition  and  Support  Pro¬ 
cedures  for  Computer  Resources  in  Sys¬ 
tems 

MIL-Q-9858A,  Quality  Program  Require¬ 
ments 

MIL-S-52779,  Software  Quality  Assurance 
Program  Requirements 

MIL-STD-1521A,  Technical  Reviews  and 
Audits  for  Systems,  Equipments  and  Com¬ 
puter  Programs 

D180-19846-1 ,  Software  Quality  Assurance 
Guide 

MIL-D-83468,  Digital  Computational  Sys¬ 
tem  for  Real-Time  Training  Simulators 

AFLC  Regulation  66-37,  Management  of 
Automated  Test  Systems 


SAMS0-STD-73-5B,  Quality  Assurance  Pro¬ 
gram  Requirements  for  Space  and  Missile 
Systems 

MIL-STD-483,  Military  Standard  Config- 
figuration  Management  Practices  for  Sys¬ 
tems,  Equipment,  Munition  and  Computer 
Programs 

SSD  Exhibit  61-47B,  Computer  Program  Sub¬ 
system  Development  Milestones 

DOD  Director  5000.29,  Management  of  Com¬ 
puter  Resources  in  Major  Defense  Systems 

DOD  4120 .17 -M.  Strategic/Automated  Data 
Systems 

DOD  Weapon  Systems  SW  Management.  Study, 
May  1975  Johns  Hopkins  University, 
Appl ied  Physics  Lab 

Defense  Systems  Software  Management 
Plan,  March  19,  1976,  Barry  De  Roze, 
0ASD 


Section  3.0  EARLY  PLANNING  CONSIDERATIONS 


To  plan  adequately  for  ATE  and  TS  soft¬ 
ware  QA,  the  USAF  P0  must  consider  deci¬ 
sions  and  directions  established  early 
in  the  the  conceptual  phase  of  the  prime 
weapon  system  acquisition  cycle.  Since 
ATE  and  TS  are  support  systems  usually 
contracted  for  separately,  or  as  an 
addition  to  the  prime  contract,  at  same 
point  during  the  full  scale  development 
phase  of  the  prime  system,  much  of  the 
software  development  methodology,  in¬ 
cluding  SWQA  will  have  been  established 
at  the  time  of  ATE/TS  contract  award. 
Usually,  QA  and  Configuration  Management 
(CM)  methodology  similar  to  that  applied 
to  the  prime  weapon  system,  will  also  be 
applied  to  the  ground  support  systems. 
The  guidebook  on  "Computer  Program  Docu¬ 
mentation  Rquirements"  CDRL  item  A009, 
discusses  ways  in  which  ATE  and  TS  sys¬ 
tems  are  acquired.  Depending  upon  the 
parti cular  contracti ng  arrangement, 
early  development  planning  may  be  done 
by  the  PO  or  by  the  contractor.  Which¬ 
ever  applies,  QA  planners  (within  the  PO 
or  within  the  contractor's  organization) 
should  support  early  system  planning 
efforts  recognizing  that  methods  of  CPCI 
development  established  for  the  prime 
weapon  ystem  may  influence  those 
applied  to  ATE  and  TS  systems. 

3.1  ASSIGNMENT  OF  RESPONSIBILITIES 

While  the  majority  of  the  Quality  Assur¬ 
ance  efforts  required  for  the  ATE/TS 
software  development  life  cycle  are 
accomplished  by  the  contractor,  QA  per¬ 
sonnel  within  the  PO  must  be  among  the 
first  concerned  with  QA  planning.  Before 
software  development  even  begins,  much 
can  be  done  toward  defining  quality  soft¬ 
ware  in  the  analysis  phase.  The  follow¬ 
ing  sections  discuss  the  coordinated 
efforts  of  government  and  contractor  QA 
personnel  as  a  function  of  acquisition 
phase.  Figure  3.1-1  illustrates  the 
major  USAF  and  contractor  SWQA  respon¬ 
sibilities  as  a  function  of  ATE/TS 


system  acquisition  phase.  Note  that  for 
ATE  systems  the  contractor  has  the  major 
task  in  preparing  system  and  CPCI 
development  specifications. 

3.2  ORGANIZING  FOR  SWQA 

Since  the  contractor  performs  the  major¬ 
ity  of  ATE  and  TS  production  SWQA  tasks, 
one  of  the  PO 1 s  prime  concerns  in  plan¬ 
ning  should  be  to  evaluate  how  the  con¬ 
tractor  is  planning  and  organizing  for 
SWQA.  Contractors  should  employ  some 
kind  of  organizational  checks  and  bal¬ 
ances  to  promote  an  objective  evaluation 
of  software  design.  For  example,  a 
separate  group  should  exist  to  evaluate 
and  test  computer  programs.  This  group 
is  charged  with  the  responsibility  for 
design  and  performance  verification. 
Another  separate  group  should  exist  to 
manage  program  configurations.  This 
group  is  responsible  for  software  identi¬ 
fication,  change  control,  problem  resolu¬ 
tion  methods,  computer  program  library 
controls,  and  release  of  all  engineering 
documentation  confi guri ng  software. 
Figure  3.2-1  represents  one  way  of  organ¬ 
izing  for  SWQA.  In  this  arrangement, 
prime  software  development  responsi¬ 
bility  lies  within  the  engineering  organ¬ 
ization  during  development  phases 
(Analysis  and  Design)  while  quality 
assurance  assumes  prime  responsibility 
for  configuration  control  and  acceptance 
during  validation  phases  (test  and  inte¬ 
gration  and  subsequent  phases).  This 
responsbi 1 ity  transfer  is  necessary  be¬ 
cause  QA  is  inherently  responsible  for 
producing  official  objective  records  of 
requirements  satisfaction.  They  must 
show  exactly  what  programs  were  tested 
to  what  procedures,  in  satisfaction  of 
what  requirements.  Prior  to  this  time, 
software  engineering  is  responsible  for 
conceiving,  developing,  and  configuring 
a  design  up  to  a  point  where  it  is 
deemed  ready  for  formal,  controlled 
testi ng. 
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Figure  3.1*1.  A  TE/TS  System  Life  Cycl&  SWQA  Responsibilities 


PROGRAM 

MANAGEMENT 


rz 

n 

EQUIPMENT 

CONFIG 

DESIGN 

MGT 

i 

SOFTWARE 

i 

i - 

L. 

SOFTWARE 

PRODUCT 

ASSURANCE 


FORMAL  TESTS 

•  ACCEPTANCE  TESTS  (OR) 

•  VALIDATION  TESTS 

•  SYSTEM  TESTS 

•  TEST  PLANS/PROCEDURES 


QUALITY 

ASSURANCE 


SOFTWARE 

QUALITY 

ASSURANCE 


DEVELOPMENT  PHASE 


DESIGN  REVIEW 
•CORRECT  LOGIC 

•  TESTABILITY 

•  REQUIREMENTS  SATISFACTION 

•  CODING  ERRORS 

DEVELOPMENT  TEST 

•  MODULE  TESTS 
INTEGRATION  TESTS 

•  INTERFACE  TESTS 


~  SWQA  PROCEDURES 
—  SPECIFICATION  REVIEW 
—  AUDIT 

VALIDATION  PHASE 

—  TEST  PROCEDURE  REVIEW 
—  VERIFY  SW  CONFIGURATION 
—  VERIFY  TEST  SETUP 
—  PLANNING  ORDERS 
—  DISCREP  REPORTING 
-  TEST  WITNESS 


SOFTWARE 

DESIGN 


—  CHANGE  ACCOUNTABILITY 

—  MASTER  MEDIA  CONTROL 
“  AS  TESTED  RECORDS 


—  REQUIREMENTS  definition 

—  PROGRAM  DESIGN 

•MODULE  DESIGN 

•  INTERFACE  DESIGN 

•  DATA  BASE 

•  SUPPORT  SW  DESIGN 

•  SIMULATION  SW  DESIGN 

•  TEST  SW  DESIGN 


SOFTWARE 

CONFIGURATION 

MCT 


—  PROGRAM  CONFIGURATION 

—  CHANGE  CONTROL 

—  PROBLEM  HANDLING/REPORTING 

—  CPL  MANAGEMENT 

—  DOCUMENTATION  CONTROL 

•  SPEC  RELEASES 

•  VDD 

•  PRODUCT  SPECS 

•  OPN  MANUALS 


Figure  3.2- 1.  Typical  SWQA  Organizational  Responsibilities 


The  program  QA  organization  shown  in 
Mgure  3.2-1  includes  a  subset  software 
QA  function,  which  draws  on  the 
resources  of  other  organizations  in 
accomplishing  program  SWQA  requirements. 
The  software  QA  function  should  be  staff¬ 
ed  by  one  or  more  QA  engineers,  prefer¬ 
ably  knowledgeable  in  both  quality  and 
computer  sciences,  who  will  integrate 
and  oversee  the  software  QA  effort  for 
the  project.  The  software  QA  engineer 
writes  the  SWQA  plan  for  the  project, 
reviews  and  audits  documentation 
releases,  audits  requirements  to  proce¬ 
dures  traceability,  and  insures  the 
smooth  integration  of  all  other  SWQA 
functions  which  require  the  resources  of 
other  organizations*  He  supports  all  pro- 
gram  milestones  -  Design  Reviews,  Audits 
Test  Readiness  Reviews,  Program  Review 
meetings,  etc*,  and  is  essentially  a 
technical  QA  staff  assistant,  acting  as 
consultant  on  software  matters  to  QA 
management  and  factory  QA  personnel. 


FWp’T?  ?f  -,hf  or?ani' nation  shown  in 
ngure  3.2-1  will  exist  for  ATE  and  TS 

as  necessary  to  integrate  the  responsi¬ 
bilities  of  UUT  contractors,  the  ATE  con¬ 
tractor,  and  weapon  system  design  organi¬ 
zations  interfacing  with  TS  system 
designers.  The  key  objectives  in  organiz¬ 
ing  for  SWQA  are  twofold: 


a.  To  accomplish  an  objective  eval¬ 
uation  of  performance  independent  of 
that  performed  by  the  design  organiza¬ 
tion. 


b.  To  arrange  for  control  of  program 
materials  and  insure  that  documentation 
conforms  to  contractual  requirements  and 
established  programming  standards  and 
conventions. 


The  following  sections  describe  SWQA 
planning  activities  as  they  would  begin 
with  contractor  receipt  of  go-ahead  for 
the  analysis  phase  of  a  ground  system. 
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Section  4.0  ANALYSIS  PHASE 


In  conceiving  QA  policy  for  a  ground  sys¬ 
tem  acquisition,  QA  personnel  within  the 
P0  are  guided  by  APR  74-1  and  AFSCR 
74-6.  These  regulations  refer  to  speci¬ 
fications  such  as  MIL-Q-9858  and 
MIL-C-45662  for  Quality  Assurance.  These 
specifications  are  not  software 
oriented.  Rather  they  define  QA  system 
requirements  for  any  deliverable  product 
and  are  imposed  on  the  prime  weapon  sys¬ 
tem  contract.  MIL-S-52799,  Software  Qual¬ 
ity  Assurance  Program  Requirements,  is 
gaining  increasing  utilization  as  a  con¬ 
tractual  requirement  on  large  embedded 
software  acquisitions,  and  is  most  appro¬ 
priate  for  ATE  and  TS  software  acquisi¬ 
tions.  Prospective  contractors,  however, 
frequently  claim  that  establishing  a 
software  QA  system,  specifically  com¬ 
pliant  with  MIL-S-52799,  increases 
acquisition  costs.  This  claim  may  have 
validity  in  certain  cases.  Por  example, 
in  ATE  acquisitions,  operating  system 
software  is  procured  as  part  of  the  ATE 
system  in  a  high  percentage  of  its  final 
deliverable  configuration  from  the  ATE 
contractor.  A  requirement  imposing 
MIL-S-52799  on  a  contractor  supplying 
proven  off-shelf  software,  could  be 
unnecessari ly  costly.  The  specification 
is  totally  appropriate,  however,  for  the 
development  of  UUT  software  and  the 
total  integrated  software  package.  It  is 
for  such  reasons  that  QA  planners  should 
become  familiar  with  system  requirements 
and  their  impact  on  software  as  early  as 
possible. 

4.1  SYSTEM  LEVEL  REVIEW 

The  analysis  phase  consists  of  con¬ 
ducting  the  analyses  and  trade  studies 
leading  to  preparation  and  approval  of 
the  computer  program  -development  speci¬ 
fications.  Preliminary  design  of  CPC  I  *  s 
is  done  concurrently  in  the  latter  part 
of  the  analysis  phase  and  is  concluded 
at  the  Preliminary  Design  Review  (PDR). 


4.1.1  System  Analysis 

System  analysis  consists  of  the  hier- 
archial  decomposition  of  system  opera¬ 
tional  requirements  and  the  allocation 
of  these  requirements  to  software,  hard¬ 
ware  and  operator  functions.  These  activ¬ 
ities  include  the  preparation  of 
functional  flaw  (or  decision  logic)  dia¬ 
grams,  top-down  design  diagrams,  system 
architecture  and  hardware/software  risk, 
benefit  and  cost  trades. 

Functional  flow  diagrams  consist  of 
detailed  logic  network  and  work  descrip¬ 
tions  of  each  major  function  to  be 
performed,  in  the  order  in  which  it  is 
performed.  The  analysis  of  these  dia¬ 
grams,  in  conjunction  with  the  frequency 
at  which  the  functions  must  be  per¬ 
formed,  will  be  used  in  the  determina¬ 
tion  of  requirements  for  concurrent 
operations,  critical  timing  relationship 
between  functions  and  the  allocation  of 
processing  resources  to  perform  the  func¬ 
tions.  Decision  logic  diagrams  contain 
the  same  information  as  functional  flow 
diagrams  with  the  addition  of  decision 
information. 

Hardware/software  trade  studies  are  con¬ 
ducted  to  ensure  the  proper  allocation 
of  functions  to  equipment  and  computer 
programs.  These  risks  and  cost  trades 
address  design  to  cost,  growth  poten¬ 
tial,  development,  integration  and  sched¬ 
ule  factors.  System  modeling  and  simula¬ 
tion  are  used  to  support  these  trades 
and  verify  the  results.  Pertinent  trade 
studies  are  documented  and  included  in 
Subsystem  Design  Analysis  Reports  sub¬ 
mitted  at  the  end  of  the  phase. 

4.1.2  Computer  Program  Requirements 
Analysis 

Requirements  analysis  is  an  iterative 
process,  repeated  with  revised  require¬ 
ments  until  a  compatible  set  is  pro- 


duced.  A  disciplined  and  systematic  docu¬ 
mentation  system  and  control  of  changes 
is  required  in  order  to  integrate  and 
correlate  the  work  among  participants 
and  throughout  the  iterations. 

Functional /performance  requirements  are 
derived  from  the  requirements  allocated 
to  software  from  system  analysis.  These 
derived  requirements  are  the  result  of 
software  from  hierarchial  decomposition 
from  the  very  general  requirements  to 
those  sufficiently  detailed  to  provide 
the  basis  for  design  and  verification. 
The  highest  levels  will  be  restatements 
of  the  allocated  system  specification 
requirements.  These  requirements  are 
further  decomposed  and  allocated  to 
CPCI.  Lower  levels  are  derived  to  take 
the  form  of  text,  graphs,  charts,  etc., 
and  include  performance,  timing  and 
environment  considerations.  The  result 
is  the  computer  program  development 
specification  which  consists  of  a  set  of 
statements  that  specify  exactly  what  the 
CPCI  will  do,  how  well  it  will  do  it, 
and  the  conditions  under  which  it  must 
perform.  Demonstrating  that  the  computer 
program  satisfies  each  of  these  require¬ 
ments  is  the  basis  for  its  acceptance. 

The  documentation  technique  includes 
specific  means  to  easily  trace  any  ele¬ 
ment  through  its  parent  element  to  the 
top  of  the  model.  Graphic  tools  that 
will  be  used  in  decomposition  include 
simple  tree  structures  showing  the  func¬ 
tional  breakdown  into  more  detailed 
requirements  and  Hierarchial-Input- 
Process-Output  (HIPO)  charts  that  show 
the  inputs,  processing  and  outputs  for 
each  element  of  the  decomposition. 

The  analysis  documentation  describes 
critical  decision  points  encountered, 
and  provide  the  rationale  supporting  the 
adopted  approach,  e.g. ,  trade  studies, 
simulations,  modeling,  etc.  It  is’  expect- 
e?.  i  V13*  conflicts  among  requirements 
will  be  exposed  by  the  analysis  process. 

A  list  of  all  conflicts  and  alternatives 
are  prepared.  The  analysis  shows  how  the 


conflicts  arose,  what  requirements  must 
be  relaxed  or  modified,  and  by  how  much, 
in  order  to  relieve  the  conflict. 

4.1.3  SWQA  Responsibilities 

SWQA  personnel  should  evaluate  the  func¬ 
tional  and  verification  requirements  of 
the  computer  program  development  specifi¬ 
cation  during  the  development /ana  lysis 
effort.  SWQA  should  review  the  requir°- 
ments  in  view  of  the  total  system 
requirements,  evaluate  the  ability  of 
the  requirements  to  be  verified  and  eval¬ 
uate  the  adequacy  of  the  verification 
requirements.  SWQA  should  assure  that 
complete,  consistent,  unambiguous,  test¬ 
able  and  up-to-date  requirements  are 
established  in  order  to  insure  the 
success  and  quality  of  the  software 
development  effort. 

SWQA  personnel  should  prepare  a  check¬ 
list  to  cover  the  surveillance  of  the 
software  requirements  analysis  phase. 
This  includes  a  list  of  criteria  for 
specific  items  to  be  evaluated,  such  as 
the  fol 1 owi ng: 

a.  Is  a  development  specification  for 
each  CPCI  prepared? 

b.  Docs  the  specification  tell  what 
the  computer  program  must  do,  how  well, 
and  under  what  conditions  and  describe 
the  environment  in  which  it  is  to 
operate? 

c.  Are  requirements  stated  singu¬ 
larly,  clearly  and  concisely  and  could  a 
reader  understand  the  specification  even 
though  he  is  not  completely  familiar 
with  terminology  peculiar  to  computing 
engineering? 

d.  Are  all  requirements  traceable  to 
or  derived  from,  a  higher  level  specifi¬ 
cation,  and  is  the  source  of  each 
requirement  or  derivation  shown? 

e.  Are  performance  requirements  for 
each  function  described  in  separate  para- 
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graphs?  Do  these  paragraphs  include 
source  and  type  of  inputs,  and  destina¬ 
tion  and  type  of  outputs? 

f.  Is  each  requirement  identified, 
stated  separately,  and  in  a  manner  that 
can  be  verified  by  test,  analysis  or 
observati on? 

g.  Are  quantitative  terms,  e.g., 
ranges,  accuracies,  tolerances,  rates, 
bondary  values,  and  limits  used  in  stat¬ 
ing  requirements,  such  as  a  specific  out¬ 
put  that  can  be  recorded  as  evidence  of 
satisfaction? 

h.  Is  the  specification  limited  to 
defining  requirements,  and  free  of 
design  solutions  or  methods  to  satisfy 
these  requi rements,  stating  what  should 
be  done  rather  than  how  it  will  be  done? 

i.  Are  all  operational  interfaces 
with  the  computer  program,  including 
both  hardware  and  software,  identified 
including  references  to  those  specifi¬ 
cations  which  define  those  interfaces? 

j.  Are  all  applicable  non-operational 
interfaces  related  to  computer  program 
support  and  code  generation  identified; 
such  as  specific  programming  language, 
compiler,  data  base  management  system, 
loaders,  other  utility  programs,  or 
unique  support  hardware?  Are  references 
to  appropriate  documentation  of  these 
interfaces  identified? 

k.  Are  all  inputs  to  and  outputs  from 
the  computer  program  identified? 

l.  Are  the  specified  requirements 
free  of  material  that  is  not  intended  to 
be  contractually  binding? 

m.  Do  the  quality  assurance  pro¬ 
visions  specify  the  method  to  be  used  to 
verify  each  performance  requirement  in 
sufficient  depth  to  provide  the  basis 
and  scope  of  the  test  plan? 


n.  Are  the  methods  to  be  used  and 
evidence  to  be  obtained  to  verify  com¬ 
pliance  appropriate  to  the  requirement 
being  verified? 

In  accomplishing  ATE/TS  system  speci¬ 
fication  reviews,  QA  personnel  should 
look  for  ambiguities,  omissions  and  con¬ 
flicts  which  make  the  job  of  defining 
the  CPCI  difficult.  Clarity  and  quanti¬ 
fication  of  the  following  factors  should 
be  examined. 

a.  Definition  of  interfaces  and  bound¬ 
aries  between  system  segments. 

b.  Capabilities  which  are  to  be  resi¬ 
dent  in  hardware  vs.  software  -  (i.e. , 
certain  routines  -  arithmetic  self-test, 
utilities,  loaders,  etc.,  might  be  resi¬ 
dent  in  mini  or  mi cro-processors,  in 
Read  Only  Memory  (ROM)  or  in  the  main 
program). 

c.  Definition  of  measurable  and  veri¬ 
fiable  performance  goals;  for  example, 
in  ATE  system  design  -  what  are  the 
design  goals  in  terms  of  cost,  through¬ 
put,  maintenance  etc. 

d.  System  design  commonality  require¬ 
ments  among  installations. 

e.  Software  transportabi 1 ity  (inter¬ 
changeability  among  machines). 

f.  System  capacities  -  data  storage 
handl ing  rates,  stimul i/measurement 
capabilities,  etc. 

g.  Deletion  of  unnecessary  design  con¬ 
straints  (i.e.,  specifying  second  vs. 
third  generation  ATE). 

Although  the  above  concerns  are  pri¬ 
marily  System  Requirements  Review  (SRR) 
and  System  Design  Review  (SDR)  issues  of 
concern  to  engineering  organizations,  it 
is  necessary  for  the  QA  planner  to 
review  system  requirements  thoroughly  so 
that  the  stringency  of  required  software 
QA  requirements  can  be  properly 
assessed,  and  to  provide  clarifications 
and  improvements  helpful  in  CPCI  develop¬ 
ment  specification  preparation. 


4.2  ESTIMATING  SWQA  REQUIREMENTS 

Early  in  the  analysis  phase,  system 
designers  will  have  allocated  system 
requirements  to  preliminary  CPCI 
development  specifications.  By  now,  QA 
planners  should  have  a  fair  idea  of  the 
number,  size,  complexity,  schedule  and 
cost  targets  for  CPC 1 1 s .  Although  tradi¬ 
tionally  not  a  QA  consideration  in 
designing  system  controls,  cost  is  now  a 
prime  design  driver  in  major  weapon  sys¬ 
tems.  QA  planners  must  therefore  justify 
the  control  methodology  they  select  for 
a  given  set  of  software.  Very  complex 
systems,  destined  for  multiple  users, 
involving  seve^l  hundred  thousand 
instructions,  multiple  CPCI 1 s *  of  rela¬ 
tively  complex  unproven  software  are 
prime  candidates  for  the  full-up  disci¬ 
plines  of  MIL-S-52779.  Simpler  systems 
require  proportionately  fewer  controls. 

Although  recent  committee  actions  and 
software  symposiums  are  working  on 
standardizing  Software  Configuration  Man¬ 
agement  (SCM)  and  QA  methods,  there  is 
no  standard  at  this  time  for  estimating 
the  SWQA  task.  Figure  4.2-1  presents 
generalized  guidelines  for  selection  of 
QA  control  elements  as  a  function  of  sys¬ 
tem  size,  complexity,  maturity  and  sched¬ 
ule.  After  ascertaining  the  level  of  con¬ 
trol  stringency  for  QA  requirements,  the 
next  step  is  the  preparation  of  a  pre¬ 
liminary  SWQA  plan. 

4.3  PREPARING  A  SWQA  PLAN 

There  is  currently  no  established 
standard  Data  Item  Description  (DID)  for 
a  SWQA  plan.  However,  a  committee  is 
meeting  quarterly  to  establish  a  SCM 
standard  including  a  DID  for  a  .SWQA 
plan.  (See  Bibliography  reference  2). 
The  Navy  NELC,  however,  has  developed  a 
Software  Quality  Control  Plan  Data  Item, 
UDI-R-11.  A  SWQA  program  is  comple¬ 
mentary  to  a  SCM  program.  Together  the 
two  programs,  documented  by  written 
plans,  implement  the  Computer  Program 
Development  Plan  (CPDP). 

A  typical  SWQA  plan  outline  based  on 
MIL-S-52779  is  presented  in  Fig.  4.3-1. 


As  stated  previously,  for  some  acquisi¬ 
tions,  particularly  ATE,  the  Operating 
System  (OS)  software  package  (control, 
support  and  test)  may  be  purchased  essen¬ 
tially  "off-shelf"  from  the  ATE  contrac¬ 
tor.  It  is  normally  inappropriate  to 
impose  a  SWQA  program  embodying  signifi¬ 
cant  development  controls  on  fully  devel¬ 
oped  and  proven  software.  The  weapon  sys¬ 
tem  contractor  and  his  associates, 
however,  must  integrate  their  software 
(UUT  test  and  adapter  programs)  with  the 
vendor  supplied  operating  system  to 
define  a  complete  deliverable  software 
package.  The  SWQA  plan  stringency,  there¬ 
fore,  is  CPCI  dependent.  An  early  deter¬ 
mination  should  be  made  as  to  which  pro¬ 
grams  are  subject  to  which  specific 

controls.  The  selected  controls, 

including  the  formal  SWQA  plan  should  be 
included  in  the  contracting  vehicle  (Con¬ 
tract  Change  Proposal  (CCP),  Engineering 
Change  Proposal  (ECP),  Statement  of  Work 
(SOW),  Contract  Data  Requirements  List 
(CDRL) ) .  The  prime  contract  then  passes 
along  appropriate  requirements  to  the 
ATE  or  TS  system  contractor.  A  brief  dis¬ 
cussion  of  these  SWQA  plan  elements 

follows.  In  subsequent  sections  of  this 

guidebook,  adaptions  and  considerations 
unique  to  ATE  and  TS  software  are  dis¬ 
cussed  for  each  of  these  general  ele¬ 
ments. 

4.3.1  Work  Tasking  and  Authorization 
Procedures 

Procedures  for  defining  and  estimating 
resource  requirements  for  SWQA  tasks  for 
described  in  this  section.  Job  descrip¬ 
tions,  responsibility  assignments,  sched¬ 
ules,  completion  dates,  methods  of  autho¬ 
rizing  work,  progress  and  status  report¬ 
ing  are  identified  in  this  section.  A 
checklist  of  typical  software  QA  func¬ 
tions  by  task  is  presented  in  Fig. 
4.3-2. 

4.3.2  Configuration  Management 

The  SWQA  plan  complements  the  SCM  plan 
by  defining  the  quality  assurance 
measures  to  be  taken  in  accomplishing 
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Figure  4.2-1.  SW04  Requirements  Versus  System  Complexity 


__ _ SOFTWARE  QA  PLAN  OUTLINE  -  REF.  MIL-S-52779 

I.  Work  Tasking  and  Authorization  Procedures 

•  Procedures  and  Schedules 
©  Work  Descriptions 

©  Status  Reports 
©  Resources  Estimates 

II.  Configuration  Management 

©  Baseline  Establishment 

•  Change  Accountability 
©  Audits 

III.  Testing 

©  Analysis  to  Determine  Testability 

•  Test  Plan/Procedure  Review 

©  Monitor  and  Certification  of  Test  Results 
©  Tests  vs.  Requirements  Traceability 

IV.  Corrective  Action 

•  Problem  Reporting  and  Measuring 
©  Trend  Analysis 

•  Corrective  Action  Assessment 

V.  Library  Controls 

•  Code  Control  and  Related  Documentation 
®  Media  Identification  and  Protection 

®  Change  Control 

VI.  Computer  Program  Design  Review 

•  Contract  Compliance 

«  Evaluation  of  Design  Logic 

VII.  Software  Documentation 

VIII.  Reviews  and  Audits 

IX.  Tools,  Technologies,  Methodologies 

X.  Subcontractor  Controls 


Figui  e  4. 3 - 1.  Typical  SIVtM  Plan  Outline 
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Software  QA  Functions 
by  Tasks 


A. 

Software  QA  Planning  for  Total  Project 

1. 

Contract  Review,  Planning,  Coordination 

2. 

Preparation  of  Software  QA  Plan 

3. 

Preparation  of  Project  and  QA  Command  Media 

4. 

Review  of  Computer  Program  Development  Plans  (CPDC) 

5. 

Review  of  Software  Configuration  Management  Plans 

6. 

Preparation  of  Configuration  Audit  Plan  (FCS/PCS  or  FACI) 

7. 

Preparation  of  Software  QA  Audit  Plan  and  Procedures 

B. 

Work  Tasking  and  Authorization  (Review  and  Monitor) 

8. 

Work  Authorization 

1 

9. 

Planning  and  Scheduling 

10. 

Performance  Measurement  and  Reporting 

•' 

C. 

Software  Standards 

11 . 

Design 

12. 

Documentation 

13. 

Programming 

4 

i 

! 

» 

D. 

Software  Documentation  Reviews 

14. 

Review  Documentation  Plans  and  Schedules  including 

Configuration  Control  Methods 

[ 

i 

15. 

SIRD  (Software  Interface  Requirements  Document)/TRD's 

16. 

Development  (Requirement)  Specifications 

| 

17. 

Product  (Design)  Specifications 

i 

18. 

Data  Base  Specifications 

i 

19. 

Interface  Specifications 

1 

1 

20. 

Interface  Control  Documentation 

i 

j 

21. 

Configuration  Drawing 

i 

\ 

/ 

22. 

Process  (System  Generation)  Document 

23. 

User  Manual 

) 

* 

24. 

Maintenance  Manual 

25. 

Version  Description  Document 

E. 

Software  Design  Surveillance 

r 

26. 

Requirements  Analysis 

* 

27. 

Prel iminary  Design 

28. 

Detail  Design 

F. 

Software  Design  Reviews 

29. 

Software  Requirements  Review  (SWRR) 

30. 

Preliminary  Design  Review  (PDR) 

31. 

Critical  Design  Review  (CDR) 

Figure  4.3-2  Typical  SWQA  Task  Breakdown  (Sheet  1  of  2) 
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G.  Software  Verification  and  Test 

32.  Review  of  Verification  and  Test  Plan 

33.  Review  Test  Procedures 

34.  Track  Informal  Tests 

35.  Attend  Pretest  Meeting 

36.  Technical  Support  of  Formal  Tests 

37.  Review  of  Test  Supports 

H.  Software  Tools  (Review  and  Control) 

38.  Identify  and  Evaluate  Automatic  Test  Tools 

39.  Tool  Validation 

40.  Configuration  Control  of  Tools 

I.  Coding  Control  (vs.  Standards) 

41.  Review  Program  Listings 

J.  Software  Conriguration  Management 

42.  Audit  Compliance  to  Software  Configuration  Management 
Procedures  and  Effectiveness  of  the  Program 

K.  Computer  Program  Library  and  Media  Control 

43.  QA  Establishment  and  Maintenance  of  CPL 

or  44.  QA  and  Engineering  Establishment  and  Maintenance  of  CPL 

or  45.  Engineering  Establishment  and  Maintenance  of  CTc  with  QA  Audit  CPL 

46.  Estaolish  Methods  and  Procedures  for  Media  Control 

47.  Audit  Implementation  and  Control  of  Media  Control 

L.  Discrepancy  Reporting,  Change  Board  and  Corrective  action 

48.  Support  Change  Board  Actions 

49.  Discrepancy  Reporting  -  Tracking,  Analysis  and  Corrective  Action 

M.  Configuration  Audits 

50.  Pre-Audit  Reviews 

51.  Formal  Qualification  Review  (FQR) 

52.  Functional  Configuration  Audit  (FCA) 

53.  Physical  Configuration  Audit  (PCA) 

N.  Software  Quality  Assurance  Audits 

54.  Audit  Software  Controls  (In-house) 

55.  Audit  Software  Controls  (Remote  Sites) 

O.  Procured  Software 

56.  Participation  in  Source  Selection 

57.  Participation  in  Pre-award  Review  and  Survey 

58.  Review  of  Procurement  Specifications 

59.  Review  of  Subcontractor  Software  QA  Plan 


Figure  4. 3  2.  Typical  SW QA  Task  Breakdown  ■  "beet  2  of  2) 
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change  accountability  ( i . e. ,  assuring 
the  change  has  been  incorporated  in  all 
applicable  media  and  properly  retested 
and  reconfigured).  While  overall 
responsibility  for  SCM  should  be  vested 
in  a  separate  group,  it  is  a  QA  responsi¬ 
bility  to  assure  the  "as  built"  meets 
the  "as  designed"  and  to  provide  appro¬ 
priate  records.  It  is  also  a  QA  responsi¬ 
bility  to  audit  the  SCM  function. 

4.3.3  Testing 

The  software  QA  plan  should  provide  for 
an  independent  and  objective  evaluation 
of  the  CPCI  design.  This  may  be  accom¬ 
plished  by  design  analysis,  (review  of 
program  design,  logic,  code),  inspection 
by  compliance  with  programming  standards 
and  by  actual  test.  This  section  of  the 
QA  plan  addresses  design  verification  by 
test,  and  how  responsibility  for  such 
testing  is  vested  organizationally. 
Developmental  vs.  formal  (acceptance  or 
validation)  testing  is  distinguished. 
With  respect  to  testing,  therefore,  the 
software  QA  program  should  provide  the 
following: 

a.  Design  analysis  to  determine  test¬ 
ability 

b.  Test  requirements,  test  plan,  and 
test  procedure  review 

c.  Witnessing  and  controlling  of  for¬ 
mal  testing 

d.  Documentation  allowing  repeat¬ 
ability  of  tests 

e.  Identification  and  acceptability 
of  support  software 

f.  Record  keeping 

4.3.4  Corrective  Action 

The  software  QA  program  should  provide 
for  prompt  detection  and  correction  of 
software  discrepancies.  Corrective  ac¬ 
tion  includes  categorization  of  software 
discrepancies,  reporting,  trend  anal¬ 
ysis,  and  corrective  action  assessment. 


4.3.5  Computer  Program  Library 
Controls 

The  software  QA  plan  should  provide  a 
controlled  repository  for  all  program¬ 
ming  materials,  media  ar,J  documentation. 
This  includes  decks,  tapes,  listings, 
flow  charts,  all  configuration  descrip¬ 
tors,  change  records,  problem  reports 
and  summaries.  This  library  should  be 
maintained  and  secured  to  prevent 
unauthorized  changes  to  controlled  docu¬ 
mentation  and  media. 

4.3.6  Design  Review 

The  software  QA  plan  should  provide  a 
function  which  assures  that  program 
design  is  well  structured  and  efficient, 
is  verifiable,  satisfies  the  functional 
requirements  completely  and  complies 
with  contractual  programming  standards 
and  conventions.  The  organization  in 
which  responsibility  for  these  review 
functions  is  vested  should  be  defined. 

4.3.7  Software  Documentation 

The  software  QA  plan  must  provide  for 
methods  of  assuring  complete  and  correct 
documentation  describing  the  software 
CPCI.  The  quality  assurance  plan  should 
assure  that  all  documentation  describing 
software  requi rements,  conf i guration, 
tests  and  analysis,  user  documents, 
change  and  problem  records  are  available 
prior  to  delivery  with  the  CPCI. 

4.3.8  Reviews  and  Audits 

The  software  QA  plan  should  define  QA 
conduct  of,  and  participation  in,  a 
series  of  planned  periodic  audits  to 
assure  compliance  with  approved  contract¬ 
or  plans,  procedures  and  standards.  Both 
formal  reviews  (Preliminary  Design 
Review  ( PDR ) ,  Critical  Design  Review 
(CDR),  Physical  Configuration  Audit 
(PCA),  Functional  Configuration  Audit 
(FCA))  and  informal  (i.e.  periodic 
audits  of  compliance  to  approved  CPDP, 
SWQA  plan,  CM  plan,  etc.)  audits  should 
be  addressed. 
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4.3.9  Tools,  Technologies  and  Methods 

The  software  QA  program  should  describe 
the  use  of  any  special  tools  or  technol¬ 
ogies  used  in  the  development  and  test 
of  computer  programs.  Analyzers,  auto¬ 
matic  test  program  generators,  special 
language  translators,  configuration  con¬ 
trol  aids,  etc.  are  among  the  software 
tools  which  should  be  addressed  in  this 
1  section. 

4.4  PREPARING  SWQA  REQUIREMENTS  FOR 

THE  RFP 

A  well  prepared  SWQA  plan  should  cover 
all  the  quality  concerns  applicable  to 
contracting  for  ATE  and  TS  software. 
However,  there  are  numerous  other  RFP 
elements  which  impact  QA  in  addition  to 
the  SWQA  plan.  One  of  the  PO's  prime  con¬ 
cerns  during  the  analysis  phase  should 
be  to  assure  that  only  necessary  and 
cost  effective  SWQA  requirements  are 
imposed  in  the  contract.  To  do  this  he 
must  first  estimate  complexity,  critical¬ 
ity  and  usage  requirements  for  the 
CPC  I  *  s  involved  and  then  consider  the 
numerous  RFP  elements  which  affect  SWQA. 
Among  those  most  applicable  to  QA  are: 

a.  Procurement  Specification.  (QA  pro¬ 
visions,  Section  4.0) 

b.  Statement  of  Work  (SOW) 

•J 

c.  CDRL 

f 

(1)  Computer  Program  Development 
Plan 

(2)  Configuration  Management  Plan 

(3)  Configuration  Index 

(4)  Version  Description  Document 
(VDD) 

(5)  Drawing  and  Specification 
Index 

(6)  Acceptance  Test  Procedure 

(7)  Delivery  Data  Package 

(8)  Deviations  and  Waivers 

(9)  User  Manual 


d.  Contract  Terms  and  Conditions/ 
Special  Provisions 

(1)  QA  System  Requirements 

(2)  Warranty/Rejection  Clause 

(3)  Inspection  at  Source  Clause 

Many  functional  disciplines  contribute 
to  the  synthesis  of  an  RFP.  Each  assures 
its  area  of  concern  is  adequately 
covered.  A  common  pitfall  in  RFP  prepara¬ 
tion  is  the  inclusion  of  redundant  or 
conflicting  requirements.  For  example, 
the  subject  of  "testing"  may  be  covered 
in  a  variety  of  places  such  as: 

a.  Data  Item  Description  (DID)  for 
CPDP 


b.  DID's  for  Test  Plans/Procedures 

c.  Section  4  of  CPCI  Specifications 

d.  Acceptance  Test  Section  of  the  SOW 

Conflicts  in  test  planning,  procedur- 
alizing,  establishing  prerequisites,  and 
reporting  data  can  and  do  exist  within 
RFP  elements  such  as  the  above.  For 
example,  the  DID  for  a  CPDP  may  require 
system  level  integration  testing  of 
CPCI ' s  as  a  prerequisite  to  CPCI  formal 
acceptance  test  whereas  the  Quality 
Assurance  Provisions  (Section  4  of  CPCI 
specification)  may  not  contain  a  similar 
requirement.  Careful  review  of  each  RFP 
element  for  consistency  with  other  RFP 
elements  is  required.  One  way  of  mini¬ 
mizing  conflicting  RFP  requirements  is 
as  follows: 

a.  Identify  the  needed  QA  control 
elements  for  subject  acquisition.  (See 
Fig.  4.2-1) 


b.  For  each  control  element  (i.e., 
configuration  management)  identify  by 
title  and  number  all  RFP  elements  which 
address  that  subject. 


c.  Create  a  cross  reference  matrix  of 
QA  control  elements  vs.  RFP  elements. 


— w. 
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d.  Review,  edit,  rewrite  or  delete 
conflicting  or  redundant  RFP  elements  as 
required. 

e.  Coordinate  recommended  changes 
with  applicable  functional  organiza¬ 
tions. 

Frequently  RFP  preparation  is  a  hastily 
accomplished  activity,  especially  in  ATE 
or  TS  acquisition.  This  is  mainly  due  to 
the  fact  that  the  people  most  qualified 
to  work  on  the  RFP  have  on-going 
responsibilities  for  the  primary  weapon 
system.  As  a  consequence,  RFP  work  must 
be  interlaced  with  these  other  duties. 
Coordination  and  dedication  to  the  RFP 
task,  therefore,  suffers  and  results  in 
poorly  prepared  requirements.  Exercises 
such  as  described  above  may  take  more  of 
a  "front  end"  investment,  however,  it  is 
well  worth  the  effort  to  prevent  costly 
consequences  of  ambiguous,  redundant  or 
conflicting  requirements. 


4.5  SUBCONTRACTOR  CONTROL 

Paragraph  4.2,  "Estimating  SWQA  Require¬ 
ments"  discuss  in  depth  the  methods  of 
determining  the  stringency  of  SWQA 
requirements  to  be  imposed  when  contract¬ 
ing  for  software  acquisition.  Fig  4.2-1 
presents  a  guide  for  selecting  disci¬ 
plines  to  impose  upon  a  given  subcontrac¬ 
tor  commensurate  with  the  size,  cost, 
schedule,  complexity  and  manageability 
of  various  software  components.  Good  sub¬ 
contracting  practices  take  those  issues 
into  consideration,  resulting  in  imposi¬ 
tion  of  only  practical  and  necessary  con¬ 
trols.  In  a  prime  contract  requiring 
MIL-S-52779,  the  contractor  should  flow 
down  to  his  software  subcontractors  only 
those  work  elements  needed  to  achieve 
overall  control  of  the  end  item.  As  men¬ 
tioned  previously,  it  would  be  unwise 
(costly)  to  demand  a  complete 
re-structuring  of  a  vendor's  software 
organization  and  methods  solely  for  the 
purpose  of  complying  with  MIL-S-52779  if 
he  is  supplying  an  essentially  off-shelf 
proven  product. 
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Section  5.0  DESIGN  PHASE 


The  design  phase  begins  with  CPCI  devel¬ 
opment  specification  (Part  I)  approval 
at  PDR  and  concludes  with  Part  II  speci¬ 
fication  approval  at  CDR.  The  CPCI  devel¬ 
opment  spec  is  used  as  the  allocated 
(functional  and  performance)  baseline 
against  which  preliminary  design  and 
validation  test  planning  can  commence. 
The  reader  is  referred  to  the  "Computer 
Program  Documentati on  Requi rements" 
guidebook  for  a  discussion  of  develop¬ 
ment  specifications  for  ATE  and  TS  soft¬ 
ware.  It  is  noted  that  a  development 
specification  is  not  necessarily  appli¬ 
cable  to  test,  control  and  support  soft¬ 
ware  for  ATE.  The  guidelines  presented 
below,  however,  assume  that  a  CPCI  devel- 
opnent  specification  is  required  for 
both  ATE  and  TS  software  as  the  basis 
for  design,  management  controls,  verifi¬ 
cation  and  configuration  management. 

Surveillance  of  the  software  require¬ 
ments  and  design  activities,  will  be  per¬ 
formed  by  QA  personnel  during  the  soft¬ 
ware  development  effort,  to  provide  an 
independent  evaluation  of  the  tasks  per¬ 
formed  with  technical  review  feedback  to 
the  designers.  In  addition,  this  surveil¬ 
lance  will  provide  a  level  of  under¬ 
standing  of  the  software  requirements 
and  design  necessary  to  evaluate 
detailed  test  plans  and  procedures. 
Accordingly  this  section  deals  with  QA 
activities  during  the  design  phase, 
primarily  design  review  tasks  supple¬ 
mented  by  a  discussion  of  software  stan¬ 
dards  and  design  and  development  tasks. 

5.1  DESIGN  REVIEW  AND  SURVEILLANCE 

Design  Review  activities  by  QA  personnel 
are  conducted  for  a  variety  of  reasons. 

a.  During  the  analysis  phase,  QA 
reviews  are  conducted  to  assure  that  sys¬ 
tem  level  specifications  provide  a  com¬ 
plete,  accurate  and  unambiguous  defini¬ 
tion  of  system  and  software  performance 
requirements.  During  this  phase  the  QA 
review  assures  that  the  allocation  of 
system  requirements  of  CPCI's  is  clear 


and  complete;  and  does  not  require 
assumptions  or  further  research  to  deter¬ 
mine  what  the  CPCI  must  do. 

b.  During  the  preliminary  design 
phase,  quality  reviews  are  conducted  to 
insure  that  the  decomposition  of  the 
CPCI  into  subfunctions  is  done  in  an 
orderly,  clearly  defined,  manageable  and 
verifiable  manner. 

c.  During  the  detail  design  phase, 
the  quality  review  is  concerned  with 
examining  program  design  for  compliance 
with  functional  requirements  and  program¬ 
ming  standards. 

d.  During  the  Test  and  Integration 
phases,  QA  activities  expand  to  include 
not  only  reviews  of  test  plans  and  proce¬ 
dures,  but  also  include  physical  methods 
for  managing  test  conduct  and  protecting 
software  configuration  integrity. 

The  following  paragraphs  discuss  design 
review  guidelines  applicable  to  the 
period  between  PDR  and  CDR. 

5.1.1  Preliminary  Design  Review 
Activities 

Preliminary  design  utilizes  the  concept 
of  hierarchial  decomposition.  The  higher 
level  requirements  for  each  CPCI  are 
broken  down  into  their  major  functional 
modules.  The  basic  structure  of  the  com¬ 
puter  program  is  defined  including  the 
methods  of  sequencing  and  controlling 
module  execution.  The  communications  pro¬ 
tocol  between  modules  is  designed,  mod¬ 
eled  and  analyzed.  A  preliminary  defini¬ 
tion  of  the  data  base  structure,  organi¬ 
zation  and  content,  is  generated.  Pre¬ 
liminary  sizing  and  timing  estimates  are 
prepared  for  each  module.  The  management 
of  these  computer  resources  is  a  con¬ 
tinuing  effort  that  should  be  addressed 
throughout  the  design  and  implementation 
of  each  CPCI. 
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Modeling  of  time-critical  functions  is 
also  continued  throughout  the  design 
phase.  Detailed  design  of  external  inter¬ 
faces  is  begun  and  preliminary  interface 
control  drawings  (ICDs)  are  prepared  to 
describe  data  rates,  interrupt  fre¬ 
quencies,  data  formats,  and  limits. 

5. 1.1.1  Documents  Generated.  Documents 
resulting  from  the  preliminary  design 
process  are  as  fol lows: 

a.  A  preliminary  product  (Part  II) 
specification. 

b.  A  design  control  document  which 
contains  the  overall  software  design 
describing  the  i nterrelati on  and  control 
between  CPCIs. 

c.  A  test  plan  covering  testing  at 
all  levels,  including  methods,  loca¬ 
tions,  equipment,  personnel  and  prelimi¬ 
nary  sequencing/scheduling. 

d.  Preliminary  ICDs. 

e.  Timing  and  sizing  data. 

f.  A  Test  Requirements  Document  (TRD) 
for  each  CPCI  (ATE  test  software). 

g.  Computer  program  development  plan 
update. 

h.  User's  Manual  (Draft). 

i.  Design  and  Coding  Standards. 

5. 1.1. 2  SWQA  Responsibilities.  During 
preliminary  design,  the  basic  design  for 
each  CPCI  is  established.  Software  QA 
personnel  will  prepare  a  checklist  to 
cover  surveillance  of  the  preliminary 
design  phase.  This  wi]!  include  criteria 
for  specific  items  to  be  evaluated,  such 
as  the  following: 

a.  Are  the  functions  allocated  to 
each  CPCI  further  allocated  to  the  .nod¬ 
ules  comprising  the  CPCI? 


b.  Are  the  methods  of  sequencing  and 
controlling  module  execution  and  the 
cormini  cation  protocol  between  modules 
designed  and  analyzed? 

c.  Are  time  and  core  space  allocated 
to  individual  modules? 

d.  Is  the  UUT  stimulus,  test 
sequence,  ancillary  data  been  specified? 

e.  Have  the  UUT  diagnostic  capabili¬ 
ties  been  specified? 

f.  Has  the  total  UUT  capability  been 
decomposed  into  subfunctions? 

g.  Does  the  TRD  meet  CPCI  development 
specification  requirements? 

h.  Are  ATE/TS  self  test  calibration 
frequencies /tolerances  specified? 

i.  Are  functional  data  and  cortrol 
flow  diagrams,  within  and  among  with 
CPCIs,  completed? 

j.  Is  a  preliminary  definition  of  the 
data  base  structure,  organization,  and 
content  generated? 

k.  Are  time-critical  functions  ana¬ 
lyzed  to  verify  that  system  operations 
timing  requirements  can  be  met? 

l.  Is  the  development  and  documen¬ 
tation  of  all  required  computational 
algorithms  completed? 

m.  Is  a  functional  description  and 
preliminary  design  for  each,  module 
prepared? 

r.  Are  timing  and  sizing  summaries 
prepared  and  periodically  updated,  and 
do  they  provide  allowances  for  growth? 

o.  Are  the  standard  naming,  inter¬ 
face,  etc.,  conventions  followed? 

p.  Is  the  basic  design  compliant  with 
the  project  design  standards? 


* 


26 


q.  Is  the  design  compatible  with 
external  interfaces? 

r.  Are  data  handling  limitations  of 
the  hardware  and  software  clearly 
publ ished? 

s.  Are  the  requirements  for  ATE  test 
software  properly  decomposed  and  quanti¬ 
fied?  (i.e. ,  requirements  for  self  test, 
operati onal  assurance/faul  t  isolat ion, 
calibration?) 

t.  Are  the  support  software  require¬ 
ments  properly  referenced  for  execution 
of  CPCI  designs? 

u.  Are  external  interfaces  well 
defined  (control,  support,  test)? 

5.1.2  Detailed  Design  Review 
Activities 

The  detailed  design  phase  will  normally 
begin  after  a  successful  preliminary 
design  review.  Although  the  emphasis 
changes  from  requirements  analysis  to 
software  design,  all  personnel  and 
analys i s/document  at i on  systems  remain 
intact.  If  at  some  point  in  the  design, 
coding,  or  test  and  integration,  it 
becomes  necessary  to  redefine/ 
redistribute  the  requirements,  the 
change  can  be  more  easily  implemented  by 
processing  it  through  the  previous 
sequence  of  development  steps  to  arrive 
at  the  current  version  with  complete  and 
updated  documentation. 

Simple  design  of  program  components  and 
relationships  between  modules  are  highly 
desirable.  Ideally,  program  components 
should  be  modularized  on  the  basis  of 
performing  a  single  logical  transfor¬ 
mation  (e.g.,  compute  SIN  X)  or  on  the 
basis  of  the  control  structure  organiza¬ 
tion.  The  simplest  method  of  passing 
data  between  program  components  is  to 
pass  only  the  data  needed  directly  to 
the  component  in  a  calling  sequence.  In 
a  large  complex  program,  this  can  result 
in  inefficiency  and  redundancy.  The  use 
of  common  data  structures  and  common  con¬ 
trol  blocks  where  the  required  data  is  a 


part  of  a  larger  collection  may  provide 
a  more  efficient  relationship  between 
modules.  In  this  case,  the  design  of  the 
common  data  base  is  an  integral  part  of 
the  design  process.  During  the  detailed 
design  phase,  the  software  and  data  base 
design  tasks  are  completed  and  docu¬ 
mented  in  sufficient  depth  to  permit 
coding.  Concurrent  with  these  tasks,  the 
initial  test  plan  document  is  updated, 
and  detailed  test  procedures  are  devel¬ 
oped  and  documented. 

5. 1.2.1  Requirements  Decomposition. 
The  detailed  design  process  consists  of 
the  decomposition  of  computer  program 
requirements  into  progress! vely  lower 
level  functions  until  the  design  is  com¬ 
pleted  and  documented  in  sufficient 
depth  to  permit  coding  to  begin.  Coding 
takes  place  only  after  a  successful  CDR. 
This  is  not  to  say  that  the  entire 
design  must  be  completed  before  coding 
begins,  since  the  CDR  may  be  conducted 
incrementally  for  completed  design 
segments.  Certain  functions  of  the  CPCI 
design  may  require  early  implementation; 
therefore,  design  of  these  functions  can 
be  completed  early  and  scheduled  incre¬ 
mentally  for  CDR. 

5. 1.2. 2  Program  Modeling.  The  design 
process  also  includes  modeling  of  design 
features  that  pose  difficult  design  prob¬ 
lems  and  critical  marginal  timing.  This 
technique  utilizes  a  simulation  of  the 
computer  program  representative  of  the 
computer  loading,  timing  and  execution 
sequences  and  pilot  programming  of  the 
troublesome  design  solution.  A  proposed 
solution  may  affect  the  computer  program 
functional /performance  requi rements  that 
were  generated  during  the  analysis, 
requiring  a  change  in  the  requirements. 

5. 1.2. 3  Software  System  Models.  During 
the  design  phase,  system  models  continue 
to  be  developed  in  a  hierarchial  manner. 
Initially,  emphasis  is  placed  on  develop¬ 
ment  cf  two  separate  models  detailing 
(1)  computer  processes,  and  (2)  data  ele¬ 
ments.  The  models  are  prepared  in  the 
strict,  structured  format  previously 
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defined.  A  most  important  feature  of 
these  models  is,  that  although  separate, 
they  are  complimentary  and  can  be  easily 
cross-referenced.  This  facilitates  a 
self-check  of  the  design  process  which 
may  be  the  technique  that  exposes  desiqn 
conflicts. 

5. 1.2.4  Data  Base  Design.  Design  of 
the  common  data  base  must  be  completed 
and  reviewed  early  in  the  design  phase 
in  order  that  it  may  be  coded  and  veri¬ 
fied  on  a  schedule  that  will  enable  the 
use  of  that  data  base  to  support  testing 
of  the  first  modules  to  complete  the  cod¬ 
ing  phase.  The  data  base  design  schedule 
must  be  consistent  with,  and  support  the 
scheduled  implementation  of,  program  mod¬ 
ules.  The  design  must  show  types  of  stor¬ 
age,  access  methods,  structure  and  pre¬ 
cise  definition  of  each  element  of  the 
data  base. 


f.  Is 
with  the 


the  data  base  design  consistent 
basic  design? 


g.  Is  the  identification,  design,  and 
specification  of  parameters,  passed, 
entries,  and  normal  and  abnormal  exits 
for  all  common  subroutines  completed? 

h.  Is  the  complete  detailed  defini¬ 
tion  and  documentation  of  software  inter¬ 
nal  and  external  interfaces  completed? 
(i.e.,  between  ATE  control,  test 
support  software) 

i.  Is  the  test  plan  updated  to 
reflect  any  changes  in  test  schedules, 
test  facilities,  and  test  requirements 
resulting  from  information  developed  dur¬ 
ing  detailed  design? 

j.  Are  all  TRD's  complete? 


5. 1.2. 5  SWQA  Responsibilities.  SWQA 
personnel  evaluate  the  detailed  design 
of  the  computer  program  and  data  base. 
SWQA  prepares  a  checklist  to  cover  sur- 
veillance  of  the  detail  design  phase, 
including  criteria  for  specific  items  to 
be  evaluated,  such  as  the  following: 

a.  Are  module  functions  allocated  to 
the  lowest  level  of  individually  identi¬ 
fiable  computer  program  components? 


k.  Is  the  detail  design  compliant 
with  project  design  standards? 

l.  Does  the  design  contain  provisions 
tor  debug,  test  and  maintenance? 

m.  Are  all  the  sources  of  ATE  test 
software  requirements  being  considered 
in  the  preliminary  Part  II  specification 
design  (i.e.,  -end-to-end,  diagnostic, 
station  self  test,  TRD  data,  circuit 
analyses,  etc.). 


b.  Is  a  detailed  design  representa¬ 
tion,  from  which  the  software  will  be 
coded  and  debugged,  prepared?  Sufficient 
detail  to  permit  direct  translation  to 
computer  program  code  and  to  derive 
requirements  for  development  testing  of 
design  is  required. 

c.  Is  the  refinement  of  data  storage 
and  timing  allocations  completed? 

d.  Is  the  detailed  definition  of  the 
common  data  base  content,  structure  and 
control  completed? 


el 


e.  Are  methods  of  accessing  data  base 
ements  defined? 


n.  Do  preliminary  Part  II  CPCI  speci¬ 
fications  contain  a  matrix  showing  trace- 
ability  of  software  requirements  to 
modes  of  verification.? 

5.1. 2. 6  Final  Considerations.  The  com¬ 

pletion  of  draft  product  specifications, 
ICO  s,  test  plans  and  TRD's,  signals  the 
actual  end  of  the  design  phase.  Theoreti¬ 
cally  all  of  the  foregoing  should  be 
available  at  CDR.  However,  much  is  fre¬ 
quently  incomplete  at  the  scheduled  CDR 
date.  The  use  of  positive  software  devel¬ 
opment  tracking  methods  (refer  to  guide¬ 
book  on  Measuring  and  Reporting  Software 
Status)  will  minimize  the  chances  of 
reaching  CDR  with  less  than  expected  pro¬ 
gress  in  software  definition.  Ideally 
the  draft  CPCI  product  baseline,  less 
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listings,  is  approved  at  CDR.  Theoreti¬ 
cally,  if  the  basic  program  design  (nar¬ 
rative  and  flow)  is  sound,  a  mechanical 
task  of  code  and  debug  is  all  that 
remains.  Unfortunately  both  ATE  and  TS 
software  are  subjected  to  a  perpetually 
changing  requirements  environment.  As 
weapon  system  features  and  UUT  capabil¬ 
ities  change,  so  does  the  ground  system 
software.  A  pitfall  resulting  from  this 
environment  is  that  original  require¬ 
ments  documents  become  old  and  obsolete; 
designers  are  forced  to  design  to  change 
paper  -  engineering  change  orders, 
revision  notices,  etc.  Updating  of 
requirements  documents  sometimes  suf¬ 
fers.  The  consequence  is,  that  instead 
of  a  given  CPCI  version  satisfying  a 
given  CPCI  Part  II  Section  3.0,  several 
versions  exist,  all  of  which  satisfy  the 
Part  I,  but  each  of  which  satisfies  a 
different  set  of  changes  to  the  Part  II 
specification.  The  specific  program 
design  then,  may  not  necessarily  be 
defined  in  Part  II  but  rather  in  myriad 
of  changes  to  it;  to  the  ICO,  to  the  TRD 
and  other  ancillary  "design  to"  docu¬ 
ments,  which  must  be  updated  to  FCA. 
Unless  QA  personnel  are  aware  of  this 
pitfall,  that  aspect  of  software  con¬ 
figuration  control  dealing  with  require¬ 
ment  satisfaction  may  be  at  worst,  com¬ 
pletely  violated,  or  at  best  be  a 
monumental  task  to  sort  out.  It  is  there¬ 
fore  important  at  this  phase  of  the  pro¬ 
ject  (completion  of  design  phase  at  CDR) 
to  assess  the  traceability  of  funda¬ 
mental  system  specification  requirements 
through  the  prime  item  development 
specifications,  CPCI  development  and 
draft  CPCI  product  specifications. 

5.2  SOFTWARE  STANDARDS 

Another  item  of  major  importance  to  SWQA 
during  the  design  phase  is  concern  for 
software  standardi zat ion.  Uniform  identi¬ 
fication,  terminology,  specification  for¬ 
mat,  methods  of  coding,  flow  charting, 
annotating  listings,  reporting  data, 
etc.,  all  fall  into  the  category  of  soft¬ 
ware  standards.  Other  guidebooks  deal 
extensively  with  documentation  standards 
which  address  specification  structure 


and  format  as  required  by  MIL-STD-483, 
490  and  others.  In  this  section  the  sub¬ 
ject  of  standards  will  be  continued  to 
include  design,  coding  and  flow  charting 
techniques  and  conventions  as  necessary 
to  achieve  understabi  1  ity  and  uniformity 
in  software  design  descriptions.  The 
standards  govern  the  work  of  all  design 
personnel  in  the  software  development 
organization.  Although  UUT  test  programs 
will  be  written  in  relatively  straight¬ 
forward  test  language,  the  basic  program¬ 
ming  standards  apply  to  the  development 
of  total  test  package  software  as  well 
as  to  TS  software.  The  USAF  acquisition 
engineer  should  ensure  that  these  stan¬ 
dards  are  reflected  in  his  procurement 
specifications  and  are  implemented  by 
the  winning  contractors. 

5.2.1  Design  Standards 

Some  of  the  design  standards  which 
should  oe  provided  are  outlined  in  the 
following  paragraphs: 

5.2. 1.1  Hierarchial  Program  Design. 
Programs  should  be  designed  in  a  hierar¬ 
chial  manner,  where  the  levels  of  the 
hierarchy  correspond  to  levels  of  con¬ 
trol  of  the  tasks  performed  by  the  pro¬ 
gram.  Each  program  proceeds  from  a 
single  starting  point  and  is  broken  down 
into  a  tree  structure  of  computer  pro¬ 
gram  components.  The  top-level  component 
contains  the  highest  level  of  control 
logic  and  decisions  within  the  program, 
and  either  passes  control  to  the  next 
level  component  or  identifies  the  next 
level  components  for  in-line  inclusions. 
This  decomposition  of  control  and  tasks 
continues  to  successively  lower  levels 
until  all  functions  within  the  system 
are  defined.  The  components  of  the  pro¬ 
gram  will  be  closed  subprograms  with  a 
single  entry  and  a  single  exit  point, 
and  of  a  size  which  can  be  reasonably 
comprehended  and  viewed. 

5. 2. 1.2  Standardized  Logic  Structure. 
Logic  structure  should  be  standardized 
in  a  manner  that  enhances  readability  of 
programs  and  reduces  intricate  logic 
that  is  difficult  to  validate  and 
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verify.  Only  closed  logic  structures 
should  be  employed  in  the  construction 
of  program  components.  Closed  logic 
structures  are  logic  structures  that 
have  a  single  entry  point  and  a  single 
exit  point.  (Logic  structures  employed 
in  the  software  design,  such  as  sequence 
logic  diagrams,  if-then-else  logic  dia¬ 
grams,  do  while  logic  diagrams,  etc., 
are  specified  in  the  standard.) 

5. 2. 1.3  Readability.  The  readability 
and  communicability  of  program  routines 
are  enhanced  by  restricting  their  size 
to  that  which  can  be  easily  viewed  and 
comprehended.  The  maximum  limit  will  be 
N  (for  example,  N:  50)  executable  source 
statements.  Exceptions  are  reviewed  on  a 
case-by-case  basis. 

5. 2. 1.4  Naming  Conventions.  Standard 
naming  conventions  are  essential  to 
understandi  ng  of  the  code  by  those 
responsible  for  its  test  and  maintenance 
and  for  facilitating  communications 
among  the  persons  responsible  for 
program  development.  Standard  conven¬ 
tions  should  be  identified  and  applied 
to  the  naming  of  variables,  data  base 
items,  modules,  and  lo wer  level  computer 
program  components.  Names  should  be  com¬ 
prised  of  alphanumeric  characters  and 
are  mnemonic  so  that  it  is  possible  to 
distinguish  a  computer  program  com¬ 
ponent’s  place  in  the  hierarchical  pro¬ 
gram  structure  by  its  name;  common  data 
base  names  are  distinguishable  from 
local  variable  names;  and  common  subrou¬ 
tines  are  distinguishable  from  module 
unique  subroutines. 

5. 2. 1.5  Data  Base  Structure.  The  com¬ 
puter  program  data  base  consists  of 
local  parameters  which  are  used  only  by 
a  single  program  component  and  global 
parameters  that  are  used  by  more  than 
than  one  program  component.  Local  param¬ 
eters  should  be  controlled  exclusively 
by  the  program  component.  Global  param¬ 
eters  should  be  maintained  in  a  data 
base  structure  that  can  be  supported  by 
the  programming  language  selected  for 
the  job.  Data  structures  and  access 
methods  will  be  specified  and  all 


addressing  of  data  base  items  will  be 
performed  only  through  the  use  of  data 
labels. 

5.2. 1.6  Subprograms.  Subprograms  that 
are  called  from  only  one  point  or  that 
operate  upon  the  same  data  at  each  call 
should  use  the  common  data  base  to 
receive  and  pass  parameters.  Subroutines 
that  operate  cn  different  data  with 
different  calls  should  receive  and  pass 
parameters  through  a  calling  sequence. 
Any  error  condition  should  be  returned 
to  the  calling  module  via  a  parameter 
list  and  calling  component  handles  the 
error  condition. 

5. 2. 1.7  Flow  Charting.  Flow  charting 
standards  are  applied  during  the  prelimi¬ 
nary  and  detailed  design  phases  of  the 
software  development  process  for  the  pur¬ 
pose  of  maintaining  consistency  among 
the  various  pieces  of  the  software  sys¬ 
tem  and  providing  a  means  of  understand¬ 
ing  the  proposed  design.  Normally 
included  in  the  flow  charting  standard 
are:  1)  flow  charting  procedure  includ¬ 
ing  direction  of  flow  (top  to  bottom, 
left-to-right) ,  level  of  detail,  use  of 
page  connectors,  etc.;  2)  a  standard  sym¬ 
bology  for  each  function,  decision,  con¬ 
nector,  etc.;  and  3)  a  standard  nomen¬ 
clature  to  be  used  in  naming  subrouting, 
constants,  variables,  etc.  Analogous 
standards  should  be  estalished  for  alter¬ 
nate  design  representations  such  as 
Program  Design  Language  (PDL)  if  used. 

5. 2. 1.8  Coding  Language.  A  HOL  should 
be  selected  and  specified  as  the  stan¬ 
dard  for  coding  of  computer  programs. 
The  use  of  assembly  or  machine  langua  ;e 
should  be  restricted  to  coding  of  time- 
critical  routines,  certain  executive 
functions,  input/output  handling,  bit 
manipulation  functions,  etc.;  where  the 
HOL  cannot  be  used  or  is  too  inefficient 
to  satisfy  performance  requi rements.  The 
percent  of  executable  code  which  must 
have  been  generated  directly  from  a  HOL 
should  be  specified.  It  should  be  noted, 
however,  that  the  real  time  requirements 
of  TS  may  require  use  of  machine 
language  for  certain  functions.  SWQA 
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should  coordinate  with  the  software 
development  organization  to  assure  that 
design  standards  are  developed,  pub¬ 
lished  and  met.  SWQA  should  review  the 
standards  to  assure  that  they  are 
adequate,  complete  and  monitor  the 
design  development  activities  to  assure 
that  they  are  available  and  are  being 
used.  All  deviations  should  be  reported 
to  the  design  organization  in  writing 
for  resolution. 

5.2.2  Coding  Standards 

Coding  from  the  design  representation 
entails  the  conversion  of  that  represen¬ 
tation  in  accordance  with  the  rules  of 
the  programming  language  used.  Applica¬ 
tion  of  a  set  of  standard  rules  is  con¬ 
ducive  to  quick  and  systematic  work.  The 
resulting  uniformity  in  the  coding 

method  is  of  particular  advantage  for 
maintenance  of  programs.  The  work  of  pro¬ 
gram  coding  has  been  in  a  state  of  tran¬ 
sition  for  a  number  of  years.  Origi¬ 
nally,  programs  could  only  be  coded  in 
machine  language;  nov/adays  the  HOL  is 
used  nearly  exclusively.  This  facili¬ 
tates  not  only  the  work  of  coding 

itself,  but  also  the  communication 
between  systems  personnel.  It  must,  how¬ 
ever,  be  noted  that  some  programming 

problems  can  best  be  solved  by  using 
assembly  language. 

SWQA  personnel  should  assure  that  coding 
and  programming  standards  are  adequate 
and  complete,  are  published,  available 
and  performed.  SWQA  should  review  the 
coding  standards  and  their  implementa¬ 

tion  to  assure  adherence  to  the 
following: 

a.  Coding  is  not  started  before  the 
detail  design  representation  is  prepared 
a  r.  J  the  programming  specification  is 
released;  and  a  system  for  identifica¬ 
tion  of  the  system  elements  and  data 
objects  is  prepared. 


b.  A  standard  for  the  use  of  naming 
conventions  and  abbreviations  for  large 
labels  and  symbolic  addresses  is  estab¬ 
lished.  Names  should  be  meaningful, 
reflecting  both  the  fun  tion  of  the  vari¬ 
able  or  label,  and  its  structural  rela¬ 
tionship  to  other  variables  or  labels. 

c.  Every  code  segment  contains  a 
single  entry  and  a  single  exit  (with  the 
possible  exceptions  of  routines  where 
algorithm  or  data  base  is  common  to  more 
than  one  extremely  similar  function, 
such  as  SIN/COS). 

d.  The  beginning  and  end  of  any  pro¬ 
gram  or  segment  is  completely  contained 
in  a  single  element. 

e.  Normally  coding  should  be  per¬ 
formed  with  one  statement  per  source 
image  line.  In  free  format  languages,  if 
there  is  more  than  one  statement  per 
line,  the  statements  must  pertain  to  a 
common  function. 

f.  Format  statements,  which  are  refer¬ 
enced  in  more  than  one  place  in  a  pro¬ 
gram,  are  grouped  in  one  area  to  sim¬ 
plify  debugging  and  maintenance. 

g.  Indentation  of  the  source  code  is 
used  to  correspond  to  the  logic  struc¬ 
ture  of  the  program. 

h.  Data  is  grouped  and  arranged  in  a 
meaningful  order. 

i.  Standardized  formats  are  used  for 
error  messages.  Cryptic  error  messages 
are  prohibited. 

j.  Inhibit  or  rigorously  control  the 
conditions  under  which  machine  language 
patches  may  be  used. 

k.  Every  code  segment  is  limited  to 
reasonable  amounts  of  logic,  easy  to 
understand,  analyze,  code  and  read. 

l.  Comments  are  used  where  necessary 
to  enhance  the  readability  and  under¬ 
standing  of  the  program.  Comments  are 
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used  frequently,  sufficient  to  make  the 
listing  readable  at  a  level  higher  than 
that  of  the  code  itself. 

m.  Loop  variables  are  not  altered 
during  loop  execution. 

5.3  SOFTWARE  VERIFICATION  PLANNING 

Software  verification  and  test  planning 
encompasses  many  elements  of  a  software 
quality  assurance  program  and  essen¬ 
tially  spans  the  total  software  life 
cycle.  Software  verification  includes 
techniques  that  are  used  to  assure  that 
the  computer  program  being  developed 
will  completely  and  correctly  meet  all 
of  its  requirements.  It  is  the  process 
of  determining  that  the  computer  program 
was  developed  in  accordance  with  the 
specifications,  that  it  satisfactorily 
performs  the  functions  for  which  it  was 
designed  and  that  it  does  not  perform 
unintended  functions.  Software  perfor¬ 
mance  and  design  requirements  which  must 
be  verified,  include  functional  require¬ 
ments,  interface  requirements  and  spe¬ 
cial  (non-performance)  requirements  such 
as  programming  standards,  program  organi¬ 
zation,  protection  of  classified  infor¬ 
mation,  etc. 

5.3.1  Verification  Modes 

Verification  includes  participation  in 
reviews,  studies  and  analysis,  and 
designing  and  conducting  tests  to  assure 
that  critical  requirements  are  being 
met.  The  goal  is  to  improve  the  quality/ 
reliability  of  the  software  and  uncover 
problems  early  in  the  development  effort 
so  that  they  can  be  corrected  at  a  much 
lower  cost  than  if  detected  at  a  later 
period  in  the  software  life  cycle.  Veri¬ 
fication  is  based  on  having  a  series  of 
documents  with  a  one-to-one  correspon¬ 
dence,  which  defines  and  shows  trace- 
ability  between  system  requirements, 
software  requirements,  software  design, 
software  code  and  software  tests.  These 
documents  include  a  system  specifica¬ 
tion,  a  computer  program  product  speci¬ 
fication  and  a  series  of  test  documents 


-  test  plan,  test  procedures  and  test 
reports.  Software  verification  is  a  cosi- 
tinuous  process  of  determing  whether  th?. 
output  of  each  phase  of  software  devel¬ 
opment  meets  all  requirements  imposed  by 
the  prior  phase. 

The  requirements  for  "formal"  verifica¬ 
tion  of  the  software  are  specified  in 
the  QA  section  of  the  computer  program 
development  specification.  This  section 
will  specify  requirements  for  verifica¬ 
tion  of  the  design  and  performance 
requirements  contained  in  the  require¬ 
ments  section  of  the  specification.  The 
verification  methods  may  include  inspec¬ 
tion  of  the  computer  program,  analysis 
of  the  computer  program  and  demonstra¬ 
tion  or  test  of  the  computer  program. 
The  methods  for  verification  of  each 
requirement,  with  a  one-to-one  relation¬ 
ship  between  performance  and  design 
requirements  and  verification  require¬ 
ments,  must  be  identified  together  with 
the  success  criteria. 

The  performance  and  design  requirements 
should  be  stated  in  quantitative  (verifi¬ 
able)  terms  with  tolerances  where  appli¬ 
cable.  The  methods  of  verification  are 
defined  below: 

a.  Test:  Execution  of  the  code  under 
controlled  conditions  to  generate  and 
record  realistic  performance  data.  This 
data  will  be  evaluated  to  ascertain  com¬ 
pliance  with  requirements. 

b.  Analysis:  Logical  or  mathematical 
processing  of  analytical  or  empirical 
data  under  defined  conditions  to  show 
theoretical  compliance  with  stated 
requirements.  Analysis  may  include  eval¬ 
uation  of  internal  control  logic  func¬ 
tions,  numerical  and  statistical  perfor¬ 
mance  of  algorithms  and  equations,  siz¬ 
ing  and  timing  parameters,  core  memory 
allocation,  priorities,  etc. 

c.  Demonstration:  Execution  of  opera¬ 
tional  or  functional  capabilities  before 
qualified  witnesses.  Instrumentation  and 
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data  recording  normally  will  be  provided 
indigenously  by  the  elements  being  demon¬ 
strated. 

d.  Inspection:  Examination  or  observa¬ 
tion  of  the  computer  program  against  the 
applicable  documentation  to  confirm  com¬ 
pliance  with  specified  requirements. 
Verification  by  inspection  may  consist 
of  visual  examination  for  the  presence 
of  desired  characteri sties  or  the 
absence  of  undesired  characteristics. 

A  major  portion  of  a  test  plan  comprises 
a  description  of  individual  verification 
tests  or  demonstrations  establishing  the 
performance  characteristics  set  forth  or 
derived  from  the  design  requirements,  as 
defined  by  the  quality  assurance  provi¬ 
sions  of  the  requirements  specif i cation 
and  the  external  interface  control  docu¬ 
ment.  In  laying  out  the  test  plan,  the 
originator  should  list  all  instances 
where  detailed  test  procedures  and  test 
reports  are  to  be  submitteM,  whether  cus¬ 
tomer  approval  is  required,  and  the 
mechanics  of  the  data  transmittal/ 
customer  approval  cycle,  v^his  is  shewn 
in  Figure  5.3-1,  which  summarizes  typi¬ 
cal  methods  and  required  documentation 
for  software  verif icat ion.  A  require¬ 
ments  traceability  cross-reference  index 
is  usually  included  in  the  test  plan, 
though  often  it  cannot  be  completed  for 
the  original  release.  This  table  cor¬ 
relates  the  quality  assurance  require¬ 
ments  with  the  specific  test  procedures 
that  will  verify  each  requirement.  This 
tabular  summary  in  5n  approved  test  plan 
document,  followed  up  by  approved  test 
procedures  and  reports,  constitutes  evi¬ 
dence  of  compliance  with  contractual  and 
quality  assurance  requirements  necessary 
for  product  acceptance. 

5.3.2  QA  Review  of  Test  Plans 

Preliminary  drafts  of  test  plans  should 
be  available  at  PDR  and  completed  drafts 
available  at  CDR.  QA  personnel  should 
assist  in  the  review  and  preparation  of 
early  drafts  to  insure  clear  and  umambig- 


uous  plans  are  documented  for  software 
acceptance.  As  previously  stated  soft¬ 
ware  should  be  treated  like  hardware 
from  the  standpoint  that  various  levels 
of  testing  are  required  leading  to  ulti¬ 
mate  acceptance.  The  major  types  of 
tests  conducted  on  hardware  Contract  End 
Items  (CEI),  consist  of  bench  level 
tests,  qualification  tests,  and  quality 
conformance  tests  (acceptance  tests). 
Software  testing  '$  similarly  structured 
with  two  significant  differences  that 
frequently  cause  confusion. 

a.  Quality  conformance  tests  or  accep¬ 
tance  tests  applicable  to  hardware  CEI's 
are  not  applicable  in  the  same  sense  to 
CPCI's.  These  tests  are  conducted  on 
each  serialized  unit  of  production  line 
hardware  to  verify  the  integrity  of  the 
build  process;  that  each  unit  has  been 
fabricated  per  drawing  and  to  verify 
workmanship.  Such  tests  are  not  appli¬ 
cable  to  CPCI's  because  acceptance  is 
based  on  different  criteria.  Once  an 
operational  program  is  cod^J,  compiled 
and  assembled  into  mac'.-ne  executable 
code  residing  on  tape  or  disk,  its 
build  process  is  complete  ar^  it  is 
ready  for  validation.  Once  the  machine 
executable  code  residing  on  a  master 
taPe  or  disk,  etc.,  is  validated  there 
is  no  need  to  re-execute  the  validation 
procedure  on  duplicate  copies.  Verifi¬ 
cation  of  duplication  is  all  that  is 
necessary  for  acceptance  of  additional 
copies  of  the  CPCI. 

Hardware  CEI  acceptance  is  based 
upon  successful  completion  of  CEI  quali¬ 
fication  and  acceptance  tests.  These  are 
CEI  level  tests,  not  system  tests.  ATE 
and  TS  software,  on  the  other  hand,  can 
only  be  validly  accepted  after  system 
level  tests.  For  example  -  an  ATE  soft¬ 
ware  CPCI  may  consist  of  four  major 
component  programs  -  control ,  support, 
station  self-test,  and  UUT  test  soft¬ 
ware.  Proper  execution  of  the  UUT  test 
program  can  only  verified  in  a  total 
system  environment  requiring  all  four 
comoonents  functioning  interactively. 
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These  differences  and  others  affect  the 
te*t  planning  process  for  ATE  and  TS 
software  in  several  ways. 

5.3.2. 1  Validation.  Ultimate  software 
acceptance  is  essentially  synonymous 
with  system  qualification.  This  implies 
that  system:  engineering  is  involved  in, 
if  not  the  prime  source,  for  software 
validation  test  planning.  Unless  clearly 
addressed  in  the  test  plan,  system  speci¬ 
fication  requirements  could  be  confused 
with  CPCI  Part  I  specification  require¬ 
ments  as  the  origin  of  validation  test 
planning  documents  and  procedures. 

5. 3. 2. 2  Documo  ;tation.  Acceptance  of 
software  is  contingent  upon  completion 
of  inspections,  analysis,  contractor 
verification  and  validation  tests,  demon¬ 
strations  (i.e.,  fault  detection  demon¬ 
stration  for  ATE)  and  ultimately  the 
user  conducted  "hands  on"  tests.  It  is 
normally  a  systems  engineering  function 
to  gather  all  such  evidence  of  require¬ 
ments  satisfaction  in  support  of  even¬ 
tual  FCA,  PCA  activity.  QA,  as  the 
acceptance  agency  of  the  contractor,  has 
a  vested  interest  in  assuring  this 
coordination  of  compliance  data  is 
gathered,  maintained  and  organized  in  an 
official  manner. 

5. 3. 2. 3  Evaluation.  Test  procedures 
are  typically  lengthy  and  complex 
requiring  intimate  familiarity  with  the 
system  and  programming  language.  Their 
efficient  generation  and  use  requires 
senior  engineering  personnel  as  test  pro¬ 
cedure  writers  and  conductors.  To  val¬ 
idly  assess  the  adequacy  of  such  proce¬ 
dures  as  satisfying  performance 
requirements,  reviews  must  be  conducted 
by  personnel  technically  knowledgeable 
in  ATE  or  TS  system  technology.  Unless 
QA  organizations  have  access  to  such  per¬ 
sonnel,  this  function  should  be  dele¬ 
gated  to  a  qualified  independent  test 
group. 

5. 3.2.4  Acceptance.  Although  LTCI's  do 
not  require  acceptance  tests  for  subse¬ 


quent  copies  of  the  same  program,  the 
CPCI  does  require  a  full  acceptance  or 
validation  test  whenever  it  is  used  with 
a  different  processor  configuration. 
This  is  frequently  a  consideration  in 
ATE  systems  in  which  the  sane  ba^ic  soft¬ 
ware  controls  different  functional  test 
stations.  Since  there  can  be  slight  dif- 
fererences  between  processors/stations 
which  affects  software,  acceptance  tests 
should  be  demanded  whenever  the  CPCI  is 
transported  for  execution  on  a  different 
station. 

A  good  test  program  requires  the  devel¬ 
opment  of  a  test  plan,  prepared  in  con¬ 
junction  with  the  software  requirements 
analysis  phase,  to  define  all  levels  of 
testing.  For  a  complex  computing  system, 
several  levels  of  testing  are  required 
to  verify  that  all  requirements  are  met: 

a.  Module,  or  computer  program 
component  testing  will  be  conducted  to 
verify  that  individual  components  have 
been  properly  coded  and  can  be  executed 
and  that  they  satisfy  corresponding  soft¬ 
ware  design  and  data  base  design  require¬ 
ments. 

b.  Intermodule  testing  is  conducted 
to  verify  that  software  interfaces 
between  computer  program  components  and 
the  executive  program  are  satisfied. 

c.  Computer  program  testing  will  be 
conducted  to  verify  that  hardware/soft¬ 
ware  interface  requi rements  are  properly 
implemented  and  software  requirements 
contained  in  the  computer  program  devel¬ 
opment  specification  are  satisfied. 

d.  System  testing  will  be  conducted 
to  demonstrate  the  operational  system 
specifications. 

All  testing  to  determine  compliance  with 
the  software  specification  requirements 
(computer  program  development  specifica¬ 
tion)  and  the  system  specification 
requi rements  should  be  conducted  by  an 
organization  separate  from  the  software 
design  group.  Such  independent  testing 
is  sometimes  referred  to  as  the  "Product 


Test"  function.  Product  Test  includes 
preparation  cf  test  plans,  test  proce¬ 
dures  and  test  reports  as  well  as  the 
test  cases  themselves  and  the  conduct  of 
the  tests.  The  only  exception  to  this  is 
the  computer  program  component  or  module 
test  to  assure  that  each  module  satis¬ 
fies  its  design  requirement.  This  test 
is  generally  developed  and  conducted  by 
the  software  designer  to  assure  that 
each  module  implements  the  design  con¬ 
tained  in  the  draft  computer  program  pro¬ 
duct  specification.  The  other  higher 
levels  of  test  should  be  planned  and  con¬ 
ducted  by  an  independent  test  or  quality 
assurance  organization  that  does  not 
have  a  vested  interest  in  the  design. 
Their  only  concern  should  be  the  testing 
against  requirements,  both  specification 
or  baseline  requirements  and  cascading 
requirements,  such  as  those  documented 
in  a  User  Manual. 

The  QA  representative  should  review  the 
test  plan  for  adequacy.  A  checklist, 
such  as  the  following,  should  be  pre¬ 
pared  and  used  to  assure  satisfactory 
completion  of  required  tasks. 

a*  The  software  products  to  be  tested 
are  identified. 

b.  The  extent  of  testing  is  adequate 
and  clearly  defined. 

NOTE 

The  problem  of  determing  test  suffi¬ 
ciency,  as  indicated  by  b.  above, 
should  be  recognized  during  the  test 
planning  phase,  i.e.,  it  is  not 

possible  to  review  test  plans/  proce¬ 
dures  to  ensure  that  all  branches 

and  logic  will  be  exercised.  Test 
planning  should  therefore  include 

the  use  of  tools  such  as  analyzers 
as  described  in  paragraph  5.4.  These 
tools  can  be  used  during  test  execu¬ 
tion  to  determine  the  branches  and 
code  exercised  during  execution  of  a 
test  case. 


c.  All  requirements  of  specification 
are  covered. 

d.  Test  support  software  and  hardware 
is  referenced  and  described. 

e.  The  roles  and  responsibilities  of 
each  organization  participating  in  soft¬ 
ware  testing  are  clearly  defined. 

f.  Methods  of  test  control  and  status 
reporting  are  adequately  described. 

g.  Test  schedules  and  locations  are 
specified. 

h.  Test  cases  are  defined  for  all 
requirements,  the  test  approach/method 
is  appropriate,  input/output  data  are 
adequately  defined,  success  criteria  is 
adequate. 

i.  The  general  procedures  for  analy¬ 
sis  of  test  results  are  given:  including 
identification  of  computer  programs  to 
be  used  for  data  reduction  and/or 
analysis. 

j.  The  test  plan  is  prepared  per  the 
applicable  documentation  standards. 

5.3.3  QA  Reviews  of  Test  Procedures 

SWQA  measures  related  to  verification 
testing  are  primarily  concerned  with  the 
adequacy  of  the  test  cases,  the  tesi, 
plan  and  test  procedures,  the  confor¬ 
mance  of  test  conduct  and  test  results 
with  test  procedures,  and  the  timely 
reporting  and  correction  of  all  software 
deficiencies.  As  a  minimum,  the  test 
plan  is  reviewed  in  detail  to  ensure 
that  the  technical  planning  and  test 
case  definition  is  adequate  and  com¬ 
plete;  that  appropriate  levels  of 
testing  are  planned.  The  test  plan 
should  identify  the  schedules,  test 
methods  and  success  criteria  as  well  as 
all  required  support  facilities,  equip¬ 
ment,  software  and  personnel.  It  should 
include  plans  for  testing  at  both 
nominal  and  extreme  conditions.  The 
individual  test  procedures  are  reviewed 
for  adequacy,  success  criteria,  callout 


of  required  configurations  of  support 
hardware  and  software  and  the  software 
under  test.  The  test  procedures  are 
reviewed  tc  assure  that  adequate  detail 
is  provided  to  clearly  define  the  test 
objectives,  test  inputs,  expected 
results,  data  recording  requirements  and 
data  reduction  requirements.  Test  proce¬ 
dures  are  reviewed  against  test  (quality 
assurance  or  verification)  requirements 
of  the  computer  program  development 
specification  and  the  system  specifi¬ 
cation  to  assure  that  all  requirements 
are  verified.  Traceability  between  veri¬ 
fication  (by  test)  requirements  and  indi¬ 
vidual  tests  should  be  documented. 

Formal  program  milestone  reviews  and 
Test  Procedure  Reviews  (TPR)  should  be 
held  for  computer  program  verification 
and  higher  level  tests  to  provide  formal 
approval  of  the  test  procedures  prior  to 
the  start  of  test.  Individual  test  proce¬ 
dures  should  be  reviewed  to  ensure  that 
all  significant  design  features  will  be 
tested  and  all  specification  require¬ 
ments  will  be  verified.  The  review  will 
establish  that  there  is  compatibi  1  ity 
between  the  test  plan  and  the  test  proce¬ 
dures,  test  inputs  are  capable  of 
testing  the  design  at  its  limits,  the 
test  will  satisfy  all  test  objectives, 
test  instructions  are  clear  and  concise, 
the  configuration  of  the  item  under  test 
is  accurately  described,  all  test  hard¬ 
ware  and  software  are  aceptable  and 
adequately  defined,  and  the  success 
criteria  is  consistent  with  the  test 
objectives  and  clearly  stated.  These 
reviews  are  conducted  by  the  test  organi¬ 
zation  and  attended  by  software  design, 
software  quality  assurance,  and  other 
interested  or  part ici pati ng  organiza- 
t ions. 

The  QA  representat i ve  should  review  the 
software  test  procedures  for  adequacy.  A 
checklist,  such  as  the  following,  should 
be  prepared  and  used  to  assure  satisfac¬ 
tory  completion  of  the  requirements. 

a.  Test  procedure  documents  are  pre¬ 
pared  per  the  applicable  documentation 
standards. 


b.  TPR*s  are  scheduled  and  held  prior 
to  test  conduct. 

c.  Test  procedure  documents  are  avail¬ 
able  as  scheduled  and  their  status  is 
accurately  reported. 

d.  Test  procedures  are  consistent 

with  the  computer  program  test  plan  and 
are  current  with  the  design  documenta¬ 

tion. 

e.  There  is  traceability  for  each 
requirement  to  the  test  that  verifies 
it. 

f.  All  significant  design  features 

will  be  tested. 

g.  Test  inputs  and  outputs  are  identi¬ 
fied  and  success  criteria  is  adequately 
def i ned. 

h.  The  sequence  of  ordered  steps  to 
be  executed  is  given  for  each  test. 

i.  Test  instructions  are  clear  and 

concise. 

j.  The  software  configuration  to  be 
tested  is  described. 

k.  The  hardware  and  test  software  con¬ 
figurations  are  identified  and  adequate 
for  the  test. 

5.3.4  Verification  Planning  Summary. 

The  independent  verification  of  software 
requirements  should  be  accomplished  by  a 
SWQA  function  which  is  separate  from  the 
software  development  function.  These  two 
functions  should  be  located  in  separate 
organizations  within  the  project  to  pro¬ 
vide  an  independent  check  and  balance  on 
the  development  effort.  The  technical 
basis  and  decisions  upon  which  the 
design,  code  and  test  are  founded  must 
be  challenged  and  feedback  provided  to 
the  software  development  function  to  cor¬ 
rect  errors  or  deficiencies.  The  per¬ 
sonnel  assigned  to  perform  these  SWQA 
functions  must  possess  technical  skills 
in  both  the  software  engineering  and  QA 
disci  pi ines. 
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Verification  of  requirements  should  be 
started  early  in  the  software  life 
cycle,  prior  to  preparation  of  the  com¬ 
puter  program  development  (requirements) 
specification.  This  allows  time  to 
review  the  system  requirements  and  their 
allocation  to  computer  programs  and  to 
plan  the  detailed  software  verification 
activities.  The  participation  in  soft¬ 
ware  verification  activities  by  software 
QA  personnel  provides  the  knowledge  and 
data  necessary  for  software  acceptance 
and  provides  a  higher  level  of  confi¬ 
dence  in  the  software  quality/ 
reliability.  This  continual  verification 
activity  directly  supports  the  formal 
software  configuration  audits  (FCA/PCA) 
by  assuring  that  the  software  documenta¬ 
tion  and  computer  program  code  are  con¬ 
sistent,  complete  and  satisfy  con¬ 
tractual  performance  and  design  require¬ 
ments.  Software  verification  is  a  major 
element  of  a  successful  SWQA  program.  It 
is  aimed  at:  assuring  high  quality/ 
reliability  software  products  which  sat¬ 
isfy  contractual  requirements;  reducing 
the  frequency  of  software  errors;  and 
reducing  the  overall  software  life  cycle 
costs. 

Section  6.0  summarizes  the  actual  imple¬ 
mentation  of  software  verification  and 
describes  the  conduct  and  controls  of 
that  activity. 

5.4  TOOLS,  TECHNOLOGIES,  and  METHODS 

A  number  of  tools,  techniques  and  method¬ 
ologies  are  available  to  assist  in  the 
development  and  assurance  of  high  qual¬ 
ity  software.  While  their  availability 
is  limited  at  this  time,  eventually  the 
number  of  these  will  expand  consider¬ 
ably.  Since  most  of  the  presently  avail¬ 
able  tools  are  for  the  use  of  the  soft¬ 
ware  developers,  the  software  quality 


engineer  should  be  primarily  concerned 
with  evaluating  the  application  of  those 
selected,  to  assure  they  are  properly 
used  to  bring  about  the  intended  improve¬ 
ments  in  software  quality.  In  the 
future,  when  a  diversity  of  such  tools, 
techniques  and  methodologies  are  avail¬ 
able,  it  will  be  necessary  to  narrow 
down  the  selection  to  those  which  will 
provide  optimum  results. 

5.4.1  Application 

Provisions  for  the  use  of  tools  and 
methodologies  should  be  identifiefd  in 
the  projects*  software  planning  documen¬ 
tation,  such  as  the  CPDP,  management 
plan  or  quality  assurance  plan.  It  is 
not  always  possible  to  identify  the 
specific  tools  at  the  time  the  planning 
is  established;  therefore,  provisions 
for  evaluating  and  using  tools  should  be 
part  of  the  initial  planning.  The 
specific  selections  and  criteria  for 
application  can  be  defined  later.  This 
planning  must  include  provisions  for 
budgeting,  staffing  and  using 
faci 1 ities. 

The  USAF  software  quality  engineer 
should  become  familiar  with  the  tools/ 
methods  and  their  uses  and  coordinate 
with  the  software  development  and  test 
organizations  to  participate  in  their 
selection  and  planning  for  application. 
He  should  assure  that  all  tools  are 
identified,  controlled  and  documented, 
and  are  suitable  for  their  intended  func¬ 
tions.  He  will  also  provide  surveillance 
over  applications.  Examples  of  tools  and 
methodologies  which  have  been  developed, 
and  nay  be  used  or  modified  for  use,  are 
discussed  in  the  following  paragraphs. 

5.4.2  Type  of  Tools 

Examples  of  the  kinds  of  tools  and  tech¬ 
niques  currently  used  in  the  industry  to 
help  assure  software  quality  are  briefly 
described  below.  USAF  acquisition  engi¬ 
neers  should  adequately  familiarize  them- 


selves  with  these  tools  to  determine  if 
they  warrant  special  attention  in  the 
RFP. 

a*  Review.  These  consist  of  reviews 
of  the  design  and  design  implementation 
at  various  points.  The  "peer  code 
review"  is  a  currently  used  practice 
primarily  associated  with  structured  pro¬ 
gramming  methods.  It  consists  of  eval¬ 
uations  by  members  of  the  coding  team 
evaluating  each  team  member*$  code  for 
both  correctness  and  assurance  of  good 
coding  practices. 

b.  Analyzers.  Analyzers  are  currently 
used  to  determine  the  degree  of  coverage 
by  test  plans/procedures  and  also  to  pro¬ 
vide  some  metrics  on  the  characteristics 
of  the  code  and  how  it  is  used.  Ana¬ 
lyzers  can  be  used  to  improve  test  cover¬ 
age  and,  also  to  some  extent,  to  detect 
deficiencies  in  the  functional  test  and 
design  rquirements.  They  also  provide 
the  programmer  with  data  to  assist  in 
optimizing  certain  aspects  of  the  imple¬ 
mentation  of  the  code. 

c.  Assertion  Checkers.  These  allow 
the  programmer  to  evaluate  the  ability 
of  the  program  to  detect  illegal  condi¬ 
tions.  This  is  done  by  providing  the  pro¬ 
grammer  the  means  for  inserting  illegal 
(out  of  specification)  conditions  and 
observing  the  response. 

d.  HOL  Debug  Tools.  These  are  soft¬ 
ware  tool s  which  assist  the  programmer 
in  debugging  code  and  in  making  the 
detection  of  errors  more  certain,  easier 
and  faster. 

e.  Top-down  Programming.  This  tech¬ 
nique  minimizes  the  need  for  discrete 
stages  of  software  integration  testing 
by  conducting  the  program  design  from 
the  top-most  specification  requirements 
and  integrating  them  with  the  levels 
immediately  below.  By  this  method,  when 
the  computer  program  has  been  fully 
coded,  it  also  has  been  tested  and 
integrated. 


f*  Proof-of-Correctness.  This  is  a 
technique  which  provides  for  the  appli¬ 
cation  of  assertions,  to  all  logic  paths 
in  the  computer  program  under  evalua¬ 
tion,  to  positively  establish  that  the 
functioning  of  the  program  cannot  be 
wrong. 

g.  Statistical  Predictions.  These  are 
methods  by  which  estimates  of  the  number 
of  remaining,  undetected  errors  can  be 
predicted  by  evaluating  the  previous 
error  history  of  the  software  during  its 
development.  Estimating  techniques  also 
include  estimations  of  thoroughness  of 
testing  by  the  rate  of  detection  of 
"simulated  faults"  in  ATE  software. 

h.  Audi  tors.  Code  auditors  are  tools 
which  facilitate  the  evaluation  of  code 
against  predetermined  coding  standards. 

i.  Conf iguration  Audit  Tools.  These 
are  methods  used  to  assure  the  validity 
of  the  computer  program  configuration 
integrity  by  selectively  tracing  changes 
to  the  configuration  of  the  computer 
program. 

j.  Automatic  Test  Program  Generators. 
Programs  which  produce  test  cases,  such 
as  coded  UUT  modeled  data  -  (circuit 
schematics  and  timing  diagrams)  pro¬ 
ducing  an  output  pattern  of  UUT 
responses  useful  in  testing  and  fault- 
isolating  a  UUT. 

k.  Automated  Configuration  Management 
Aids.  Programs  which  provide  automatic 
file  cataloguing,  conf i gurat i on  verifi¬ 
cation  to  assure  valid  test  results, 
file  protection  against  unauthorized 
change,  and  program  comparators  which 
detect  differences  in  programs  from  the 
basel i ne. 

l.  Media  Converters.  Programs  which 
provide  automatic  duplication,  conver¬ 
sion  and  verification  of  data  from  one 
media  (cards  to  mass  storf'  mass  store 
to  tape,  disc)  to  another. 

m.  Automatic  Tech  Order  Generators. 
Programs  which  accept  and  process  file 
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listings  of  Abbreviated  Test  Language 
for  all  Systems  (ATLAS)  test  programs, 
program  boiler  plate,  UUT  parameter 
file,  UUT  part  number  and  produces  a 
properly  formatted  USAF  Tech  Order. 

n.  Interactive  Text  Editors.  A 
program  mai  ntenance  tool  which  allows 
the  user  to  construct  and  modify  program 
source  code  or  other  textual  informa¬ 
tion. 

y  of  the  above  automated  develop- 
n.-j-'al  maintenance,  and  configuration 
control  aids  are  available  from  the  ATE 
or  TS  system  contractor  as  a  part  of  his 


basic  operating  system.  Use  of  such 
tools/system  features  can  reduce  soft¬ 
ware  development  time  significantly  over 
manual  methods.  Contractor  SWQA  per¬ 
sonnel  should  be  sufficiently  aware  of  a 
vendor's  operating  system  capability  in 
order  to  make  judgements  as  to  the 
validity  of  allowing  use  of  such  tools 
(such  a$  media  converters  or  text 
editors)  as  official  configuration  con¬ 
trol  or  validation  aids.  The  reader  is 
referred  to  Section  7.0,  Bibliography 
references  3  and  4  for  a  further  dis¬ 
cussion  of  software  program  develop¬ 
mental  and  QA  tools. 
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Section  6.0  CODE  &  CHECKOUT,  TEST  &  INTEGRATION,  AND  INSTALLATION  PHASES 


The  activities  discussed  in  this  section 
cover  the  period  from  CDR  through  deliv¬ 
ery.  It  is  during  these  phases  that  the 
as-built  software  conf i gurat ion  is  estab¬ 
lished,  checked  jnd  validated.  It  estab¬ 
lishes  whether  disciplined  design  and 
analysis  phases  have  been  completed; 
whether  the  requirements  have  been  pro¬ 
perly  defined,  interpreted  and  allocated 
to  software;  and  whether  requirement 
changes  are  minimal.  If  so,  coding, 
debugging  and  validation  efforts  should 
be  free  of  major  redesign  set-backs. 
Unfortunately,  it  is  not  realistic  in 
software  development,  especially  in  sim¬ 
ulation  and  test  software  to  expect  no 
changes.  Technologies  are  too  dynamic. 
Improvements  are  constantly  being  made 
to  the  weapon  system  performance,  accu¬ 
racies,  etc.  These  changes  correspond¬ 
ingly  affect  simulation  and  ATE  soft¬ 
ware.  Therefore,  software  developers  and 
QA  must  plan  for  efficient  change  manage¬ 
ment.  The  guidebook  on  "Software  Config¬ 
uration  Management"  CDRL  item  A00E, 
addresses  the  totality  of  change  manage¬ 
ment  for  ATE/TS  software.  This  section 
is  limited  to  a  discussion  of  those  QA 
activities  necessary  to  verify  software 
baselines  and  account  for  controlled 
changes.  The  four  areas  of  QA  activity 
which  support  the  verification  of  soft¬ 
ware  as-built  configurat i on  are: 

a.  Conf iguration  verification  &  sta¬ 
tus  accounting 

b.  Library  and  software  media  control 

c.  Control  of  formal  testing 

d.  Formal  configuration  audits 

The  following  paragraphs  discuss  QA  con¬ 
duct  of,  or  support  to,  these  functions. 


6.1  CONFIGURATION  VERIFICATION  AND 
ACCOUNTING 

The  subject  of  conf i guration  verifica¬ 
tion  and  accounting  is  used  in  this  con¬ 
text  as  meaning  all  activities  which  sup¬ 
port  the  verification  of  software  config¬ 
uration  baselines  as  modified  by  con¬ 
trolled  and  approved  changes.  There  are 
fundamental ly  two  types  of  documentation 
required  by  QA  to  achieve  conf i gurat ion 
verification: 

a.  released  engineer  documentation 

b.  manuf?cturing  planning  and  account¬ 
ing  docume  .at ion 

The  following  sections  describe  how  QA 
uses  this  documentati on  to  verify  soft¬ 
ware  configuration. 

6.1.1  Engineering  Conf igurat ion  Defini¬ 
tion  Documentation 

The  basic  configuration  description  docu¬ 
ment  for  CPC  1 1 s  is  the  Product  Specifica¬ 
tion  Part  II  per  Mil  Standard  483.  It 
includes  logic  flows,  listings  and  narra¬ 
tive  descriptions  of  program  operation. 
It  is  theoretical  ly  approved  at  CDR  as 
the  computer  program  product  baseline. 
Other  documents, ,  however,  are  required 
to  define  the  conf igurat i on  of  each  ver¬ 
sion  of  a  CPCI  program  as  it  progresses 
through  formally  controlled  development 
and  testing  phases.  The  VDD  is  an  exact 
description  of  each  individual  version 
of  a  CPCI  and  is  used  to  keep  track  of 
detailed  changes  incorporated  during  pro¬ 
gram  development.  It  is  a  modular  index 
of  subroutines  comprising  a  given  ver¬ 
sion  and  will  list  all  functional  pro¬ 
gram  components,  media,  labeling  require¬ 
ments  support  software,  utilities,  vali¬ 
dation  and  installation  instructions  nt.- 
essary  to  build,  accept,  and  load  c id 
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verify  the  machine  executable  programs 
(object  code)  into  the  designated  target 
computer.  It  is  equivalent  to  an  Line 
Replaceable  Unit  (LRU)  top  assembly 
drawing. 

Some  contracts  may  employ  equivalent 
documentation  schemes  such  as  the  mile¬ 
stone  system  per  SSD  Exhibit  61-47B. 
This  system  substitutes  8  milestone  docu¬ 
ments  which  accomplish  the  intent  of  MIL- 
STD  483  Part  1,  2,  requirements  while 
providing  a  more  granular  documentation 
structure  that  accomodates  better  func¬ 
tional  separation  of  responsibilities. 
In  such  a  scheme  the  document  equivalent 
to  the  VDD  is  the  milestone  8. 

6. 1.1.1  Ancillary  Software  Products. 
Some  ancillary  software  products  are  nec¬ 
essary  in  the  development  and  acceptance 
of  CPCI  programs,  such  as: 

a.  Automatic  Test  Program  Generators 

b.  Mission  data  programs 

c.  Memory  load/dunp  routines 

d.  Patch  loader  programs 

e.  Test/simulation  programs 

f.  Data  reduction  programs 

g.  Diagnostic  programs 

h.  Vendor  0/S  software 

These  programs  may  be  components  of  a 
deliverable  CPCI  or  CPCI ' s  themselves. 
Usually  they  are  neither.  Nevertheless, 
they  are  used  in  acceptance  activities 
for  deliverable  CPCI  * s  and  must  be  con¬ 
figured  and  accounted  for. 

6. 1.1.2  Documentation  Accountability. 
Whichever  documentation  system  applies, 
the  following  program  features  should  be 
configured,  verified  and  changes  thereto 
accounted  for  as  governed  by  program 
requirements  and  complexity. 


a.  A  narrative  program  description 

b.  A  logic  program  flow 

c.  A  source  code  program  listing 

d.  Applicable  support  software 

e.  An  object  code  program 

f.  Acceptance  or  validation  require¬ 
ments 

g.  User  instructions 

Configuration  status  accounting  docu¬ 
mentation  is  the  means  through  which 
actions  affecting  CPCI 1 s  are  recorded 
and  reported  to  program  and  functional 
managers.  It  principally  records  the 
“approved  configuration  baseline"  and 
the  implementation  status  of  changes  to 
the  baseline.  In  this  way,  it  provides 
managers  with  confirmation  that  change 
decisions  are  being  implemented  as 
directed.  Configuration  status  account¬ 
ing  is  the  recording  and  reporting  of 
the  information  that  is  needed  to  manage 
configuration  effectively.  It  includes 
listing  the  approved  configuration  iden¬ 
tification,  the  status  of  proposed 
changes  to  configurations,  and  the  imple¬ 
mentation  status  of  approved  changes.  It 
involves  maintaining  and  reporting  the 
status  of  CPCI  specifications,  asso¬ 
ciated  documents  and  proposed  changes. 
The  system  utilizes  two  primary  reports; 
the  Configuration  Index  and  the  Change 
Status  Report. 

The  computer  program  configuration  index 
provides  the  official  listing  of  the 
CPCI  specifications,  drawing  and  other 
significant  support  documents.  It  identi¬ 
fies  all  approved  changes  and  shows  the 
current  status  of  all  CPCI  documentation 
such  as  tne  computer  program  development 
and  product  specifications,  test  plans/ 
procedui  cs/^eports,  handbooks,  manuals 
and  the  VDD.  The  change  status  report 
lists  all  proposed  changes  to  the  CPCI 
documentation  listed  in  the  configura¬ 
tion  index.  It  provides  information  on 
the  current  status  of  the  CPCI  and 
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changes  throughout  its  development.  QA 
uses  the  configuration  index  and  change 
status  report  for  change  accountabil i ty 
to  assure  that  all  Class  I  changes 
incorporated  into  the  software  have  been 
approved  by  the  customer  and  that  no 
unapproved  Class  I  changes  are  incorpo¬ 
rated. 

The  period  during  which  engineering 
defined  configuration  description  docu¬ 
ments  must  be  formally  controlled,  typi¬ 
cally  begins  with  CDR,  or  completion  of 
intermodule  testing.  This  signals  the 
completion  of  developmental  testing 
during  which  engineering  controls  soft¬ 
ware  configuration  internally.  As  stipu¬ 
lated  in  the  CPDP  and  CM  plans,  QA 
begins  at  this  point.  QA  controlled 
tests  are  variously  called  Product  Assur¬ 
ance  tests,  Contractor  Internal  Verifica¬ 
tion  test,  System  Validation  tests,  or 
simply  acceptance  tests.  Paragraph  6.4 
discusses  the  attributes  of  formal  QA 
controlled  tests.  At  this  point,  how¬ 
ever,  let  us  examine  the  non-engineering 
documentation  required  to  accomplish 
CPCI  configuration  verification  and  sta¬ 
tus  accounting. 

6.1.2  Manufacturing  and  QA 
Documentation 

In  addition  to  released  engineering  docu¬ 
mentation,  there  are  numerous  manufactur¬ 
ing  and  QA  documents  and  records  which 
must  be  initiated  to  control  and  monitor 
software  configuration.  The  USAF  role 
here  is  to  understand  how  contractor  QA 
personnel  assure  that  the  as-tested  con¬ 
figuration  is  controlled  during  testing, 
and  how  changes  in  program  configuration 
are  accounted  for  to  insure  the  "as- 
delivered"  program  configuration  matches 
the  ' as-approved. "  USAF  personnel  should 
periodically  audit  these  contractor  func¬ 
tions  to  insure  objective  records  of  pro¬ 
gram  configuration  accountabi 1 ity  and 
test  are  available  to  support  the  submit¬ 
ted  delivery  documentat ion.  The  follow¬ 
ing  describes  the  function  of  the  major 
manufacturi ng  and  QA  documents. 


6. 1.2*1  Test  Planning  Order.  This 
order  defines  the  step  by  step  process 
by  which  the  software  product  is  to  be 
built  (i.e.,  assembled  and  compiled,  and 
linked  if  applicable)  tested,  dupli¬ 
cated,  verified,  labeled  and  acceptance 
stamped.  It  defines  in  extremely  precise 
detail  the  sequence  of  operations 
required  to  formally  build,  test  and 
accept  the  applicable  program. 

A  typical  format  for  a  software  verifica¬ 
tion  or  planning  order  is  shown  in 
Figure  6.1-1.  Although  simplified  this 
figure  demonstrates  the  orderly  fashion 
in  which  formal  software  testing  is 
accomplished.  Its  objective  is  to  pro¬ 
vide  a  record  which  leaves  no  doubt  as 
to  exactly  what  was  tested  and  how  such 
tests  were  conducted.  It  assures  all  the 
necessary  engineering  drawings  and  test 
procedures  are  released  authorizing  what 
to  test  and  how  to  test  it.  It  assures 
all  operations  are  completed  satisfac¬ 
torily  before  closure  of  the  order  and 
release  of  the  software  under  test  as  an 
accepted  product. 

6. 1.2. 2  Discrepancy  Report  (DR).  This 
report,  sometimes  called  a  rejection 
report,  unplanned  event  report,  material 
review  report,  pick-up,  etc.  is  the  con¬ 
tractor’s  official  form  for  reporting 
product  discrepancies.  It  is  used  to 
document  a  physical  product  deficiency, 
nonconformance,  a  procedural  error,  a 
planning  error  or  any  non  planned  event 
which  impacts  the  product  bui 1 d-test-and- 
accept  cycle.  It  must  be  satisfactorily 
dispositioned  prior  to  closure  of  the 
test  planning  order.  In  software  test¬ 
ing,  it  is  used  principally  to  document 
changes  in  program  configuration  such  as 
machine  language  "patches"  needed  to 
incorporate  make-work  changes  to  the 
program.  In  this  capacity,  it  serves  as 
an  interim  configuration  descriptor  for 
the  software,  but  since  it  is  not  offi¬ 
cially  released  engineering,  it  must  be 
held  open  until  an  engineering  change  is 
released  to  define  the  configuration 
departure  made  via  the  DR. 
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Figure  6. 1 - 1.  Formal  Test  Planning  Order 


6. 1.2. 3  Planning  Accountabi 1 i ty 

Record.  A  planning  accountabi 1 ity  record 
(PAR),  or  equivalent,  is  required  to 
insure  that  all  planning  orders  written 
to  implement  an  engineering  change  are 
accounted  for  and  completed  in  a  given 
product.  Software  is  no  exception.  For 
example,  a  given  change  might  require: 

a.  Use  of  a  new  compiler 

b.  Merge  of  two  CPCI's  into  one 

c.  Added  functional  capability 

d.  Installation  into  a  new  loading 
devi ce 

Accomplishing  this  change  on  a  given 
CPCI  might  require  4  separate  planning 
orders,  one  for  each  of  the  above  func¬ 
tions.  Separate  orders  are  required 
because  of  concurrent  activity  at  differ¬ 
ent  locations.  A  PAR  insures  all  work  is 
completed  prior  to  change  closure. 

6. 1.2. 4  Conf iguration  Summary.  This 

document  is  prepared  by  contractor  QA 
from  their  completed  planning  order 

records.  A  separate  configuration 
summary  is  prepared  for  each  serialized 
end  item  to  accurately  describe  change 
incorporation.  It  defines  all  the  Class 
I  and  Class  II  changes  incorporated  in 

the  product  since  its  creation.  For 
software,  it  is  vital  to  insuring  knowl¬ 
edge  of  configuration  for  a  product 

embodying  numerous  changes. 

The  foregoing  are  the  main  non- 
engi neeri ng  documents  and  records 
required  by  QA  to  maintain  configuration 
status  accounting.  Figure  6.1-2  shows 
the  sequence  of  events  involved  in  using 
these  documents  to  track  software 
program  configuration. 

6.1.3  Verifying  Program  Conf i guration 

Unlike  hardware,  which  is  physically 
inspectable,  the  job  of  verifying  pro¬ 
gram  configuration  is  quite  different. 
It  is  neither  practical  nor  necessary  to 
physically  inspect  a  machine  language 


program.  Rather,  the  functional  config¬ 
uration  of  a  piece  of  computer  sensible 
media  is  validated  and  then  that  media 
is  identified  and  controlled.  From  the 
master  tape  (or  disk),  duplicates  are 
made  and  verified  using  utilities 
designed  for  that  purpose.  Multiple 
copies  of  the  master,  plus  hard  copy 
listings,  preserve  the  validated  program 
configuration.  If  any  changes  (machine 
language  patches)  are  needed  to  validate 
a  given  program,  those  patches  must  be 
defined  by  released  engineering  and  con¬ 
figured  on  the  VDD.  In  the  hardware 
world,  acceptance  is  based  upon  first 
inspecting  the  article  to  insure  it  has 
been  built  per  print  followed  by  accep¬ 
tance  tests  to  prove  its  functionality. 
For  software,  acceptance  is  based  on 
functional  tests  supplemented  by  inspec¬ 
tion  and  analysis  of  program  listings. 
Since  acceptance  by  analysis  involves 
examination  of  a  source  program  listing, 
flow  and  narrative  for  theoretical ly 
correct  functioning,  the  assumption 
exists  that  the  examined  source  code  has 
been  properly  assembled  compiled,  link 
edited,  etc.  in  accordance  with  function¬ 
ally  proven  support  software  and  docu¬ 
mented  procedures,  such  that,  the 
machine  executable  program  could  be 
recreated  from  the  source  code  and 
support  software  as  defined  by  drawing. 
The  ultimate  objective  of  program  config¬ 
uration  verification  is  to  insure  the  as 
built  and  tested  program  conforms  to  the 
approved  baseline  plus  approved,  con¬ 
trolled  changes.  In  paragraph  6.4  we 
shall  examine  some  actual  techniques  for 
controlling  software  configuration 
during  test. 

6.2  LIBRARY  CONTROLS 

Computer  Program  Library  (CPL)  controls 
may  be  implemented  in  a  variety  of  ways. 
They  may  be  established  and  controlled 
by  the  SCM  organization,  QA  or  engineer¬ 
ing.  Although  strongly  supportive  of  con¬ 
figure'  :on  management,  the  requirement 
for  library  controls  is  specifically 
required  by  MIL-S-52779,  and  hence  is 
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Figure  6.  bZ  Forms I  Software  Test  Control  Responsibilities 


Documentation  Control 


discussed  in  this  guidebook.  CPL  con¬ 
trols  are  essential  for  the  following 
reasons: 

a.  To  assure  that  programs  and  data 
are  properly  identified 

b.  To  prevent  damage,  degradation, 
loss  or  unauthorized  change 

c.  To  facilitate  transfer  of  data  to 
~omputer  memory 

A  good  CPL  operation  should  provide  for 
organized  and  protected  storage  and  main¬ 
tenance  of: 

a.  Program  documentation 

b.  Source  and  object  program  materi¬ 
al  s 

c.  Change  records  and  authorizations 

d.  Problem  reports  and  corrective 
action  records 

e.  Support  programs 

The  CPL  should  be  readily  accessible  to 
the  programming  and  test  groups.  The 
library  function  should  be  thoroughly 
documented  and  audited  by  QA.  A  detailed 
CPL  procedure  should  be  released  and 
maintained  current  so  that  users  can 
understand  how  their  materials  are  orga¬ 
nized  in  the  library  and  what  they  must 
do  to  access  such  materials.  Since  one 
of  the  prime  purposes  of  the  library  is 
to  prevent  un  ~ized  change  to  media 
or  document'  the  CPL  procedure  must 

describe  or  reference  the  process  by 
which  previously  approved  source  code 
lists,  decks  or  object  lists  nay  be 
changed.  The  CPL  function  should  be 
established  to  smoothly  integrate  the 
functions  of  media  management,  change 
management,  problem  reporting  and  status 
monitoring.  Records  and  inventories 
should  be  maintained  of  all  active  and 
inactive  program  versions  both  validated 
and  unvalidated.  Let  us  now  examine  jome 
of  the  details  involved  in  operating  a 
CPL. 


6.2.1 

Formal  released  engineering  documenta¬ 
tion  central  is  not  a  function  of  a  CPL, 
but  rather  of  a  formal  central  Engineer¬ 
ing  Release  Unit  (LRU).  CPCI  listings 
(source  and  object)  he  ;ever,  are  not  nor¬ 
mally  considered  part  of  the  CDR 
approved  engineering  ba.  .line  and  there¬ 
fore  must  be  maintained  under  internal 
control  (CPL  control)  until  validated. 
This  is  the  only  practical  way  of  con¬ 
trolling  the  configuration  of  program 
code.  CPL  controlled  documentation 
includes  such  listings  and  other  docu- 
mentation  (data  base  descriptions,  load 
maps,  etc.)  needed  to  configure  the 
applicable  CPCI  version  for  eventual 
formal  release  in  the  VDD.  This  sort  of 
documentation  must  be  maintained  under 
strict  control  by  the  CPL.  For  example, 
consider  a  module  of  a  TS  CPCI.  Upon  com¬ 
pletion  of  module  verification  tests  by 
the  programmer,  the  module  is  submitted 
to  the  CPL.  At  this  point,  the  program¬ 
mer  can  make  no  more  changes  to  his  pro¬ 
gram  without  authorization.  This  is  to 
insure  that  other  programmers  w^o  may  be 
designing  interfaces  with  his  'dule  can 
do  so  with  predictable  re  's.  The 
source  code  for  the  module  may  still 
require  modificaton  after  submission  to 
the  CPL,  ho/ever  it  is  done  in  a  con¬ 
trolled  fashion  by  obtaining  review  and 
approval  by  a  designated  authority.  The 
change  request  and  approval  cycle  for 
such  a  change  is  dealt  with  in  depth  in 
the  Configuration  Management  guidebook. 
The  only  point  being  made  here  is  that 
source  code  changes  reed  to  be  con¬ 
trolled  while  not  requiring  formal  ERU 
authority  until  the  formal  test  phase* 
Management  of  such  changes  is  a  function 
of  SCM  internal  change  control  and  the 
mechanics  of  implementation  is  a  CPL 
function. 

6.2.2  Program  Media  Control 

This  subfunction  of  the  CPL  is  concerned 
with  organization  and  protection  of  the 
computer  sensible  meoia.  Loss  of  data 
due  to  defective  media  can  be  disas¬ 
trous.  Included  in  the  category  of 
“media  control"  is: 
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a.  Storage  and  protection  of  tapes, 
decks,  discs  etc. 

b.  Media  conversion  materials  and  pro¬ 
cedures 

c.  Media  duplication  and  verification 
procedures 

d.  Media  identification  and  revision 
systems 

e.  Media  certification  (free  of 
defects)  methods. 

The  objective  of  media  control  is  data 
retrievability  and  configuration  con¬ 
trol.  Decks  that  are  disorganized, 
unidentified,  mutilated  etc.  cause 
serious  data  losses  and  program  delays. 
By  establishing  a  centralized  CPL  func¬ 
tion,  the  programmer  is  relieved  of  the 
burdens  of  media  control.  Tapes  which 
will  not  load  because  of  skew,  noise  or 
other  permanent  tape  defects  can  cause 
critical  schedule  slides.  Data  which  has 
been  unsuccessfully  converted  from  one 
media  to  another  is  another  source  of 
aggravation.  Much  is  available  in  the 
literature  on  the  quality,  care  and  hand¬ 
ling  of  magnetic  storage  media  (Bibliog¬ 
raphy  references  7  through  9).  The  main 
environmental  and  handling  considera¬ 
tions  for  computer  program  storage  media 
are  summarized  below. 

a.  Temperature,  humidity  and  air 
cleanliness  data  should  be  specified  for 
the  media  storage  facility. 

b.  Periodic  inspections/audits  should 
be  conducted 

c.  Storage  racks,  bins,  and  con¬ 
tainers  should  conform  to  accepted  prac¬ 
tices. 

d.  The  Library  should  be  a  controlled 
access  area. 

e.  Smoking  should  be  prohibited. 

f.  Magnetic  tape  should  be  supported 
by  the  hub,  not  stacked  horizontally. 


g.  A  media  sign-out  file  should  be 
maintained  for  tape  copies.  The  master 
should  not  be  released  after  validation. 

If  the  QA  organization  does  not  run  the 
CPL  operation,  it  must  adopt  some  method 
of  controlling  the  validated  object  pro¬ 
gram  media.  In  some  organizations,  a 
separate  protected  area  is  designated 
for  storage  of  "QA  master"  tapes  which 
have  been  or  are  undergoing  validation. 
This  area  is  not  part  of  the  CPL,  how¬ 
ever  it  supplements  the  CPL  function  by 
creating  a  QA  controlled  area  for 
"working  copies"  of  validated  media. 
This  area  is  typically  called  a  “QA 
Master  media  file"  and  is  a  repository 
for  all  QA  controlled  software,  not  only 
ATE  or  TS  CPC  I  items,  but  avionics, 
test,  instrumentation  etc.,  type  soft¬ 
ware.  It  is  advisable  to  categorize  soft¬ 
ware  in  the  CPL  and  in  the  QA  master 
media  file  in  groups  as  follows: 

Validated  -  Active 

Validated  -  Inactive 

[Invalidated  -  Active 

Unvalidated  -  Inactive 

Validated-act ive  media  are  those  which 
are  authorized  for  usage  in  current 
tests  or  mission  simulations.  Unvali  — 
dated-active  are  those  currently  under¬ 
going  validation,  and  until  testing  is 
successfully  completed,  the  master  of 
the  baseline  med^a  undergoing  a  test 
must  be  maintained.  Unval idated-inactive 
media  sometimes  results  when  a  given 
tape  version  begins  a  test  and  then  is 
superceded  by  a  new  compilation.  If  part 
of  the  test  is  completed  with  the  origi¬ 
nal  version,  that  version  must  be 
retained  as  part  of  the  planning  order 
records.  This  situation  is  extremely 
rare  and  highly  undesirable  because  sat¬ 
isfaction  of  a  given  test  with  two  or 
more  different  software  versions  creates 
the  problem  of  having  to  identify  ver¬ 
sion  differences  and  prove  why  certain 
test  functions  completed  with  the  origi¬ 
nal  version  need  to  be  re-run  with  the 
new  version.  The  objective  of  this  file 
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is  to  retain  all  versions  of  software 
used,  either  partially  or  totally, 
during  the  conduct  of  a  given  formal 
test, 

6,2,3  Change  and  Discrepancy  Record 
Control 

There  are  two  basic  sources  of  program 
changes  -  design  changes  and  problems  or 
deficiencies,  A  detailed  discussion  of 
change  mechanics  is  reserved  for  the 
guidebook  on  “Configuration  Management". 
Tne  CPI  function,  however  must  provide 
for  maintenance  of  the  physical  media 
and  documentation  change  records.  Any 
change  processing  system  must  require 
some  formalized  record  for  identifying, 
describing  and  authorizing  changes. 
Forms  designed  for  software  internal 
change  control  are  typically  called  a 
Design  Change  Request,  Software  Problem 
Report  (SPR)  or  variation  thereof.  How¬ 
ever  desiqned  or  named,  these  forms  must 
be  numbered,  dated  and  contain  the 
following  information: 

a.  Name  of  originator 

b.  Reason  for  and  class  of  change 

c.  Problem  description 

d.  Proposed  solution 

e.  Type  of  problem  -  coding,  design, 
data  base,  computational  logic 

f.  Versions  and  documentation  updated 

g.  Organization/person  responsible 
for  fix 

h.  Approval  authority 

i.  Version  fix/change  incorporated  in 

j.  Retest  requirements 

The  CPL  must  provide  for  retention  and 
organization  of  whatever  materials 
describe  these  change  parameters. 


6.2.4  Support  Software  Control 

In  order  to  be  able  to  trace  the  origin 
of  any  given  program  version  to  its 
source  materials,  all  of  the  support 
materials  needed  to  produce  a  machine 
executable  load  module  must  be  identi¬ 
fied  and  maintained  in  the  CPL.  This 
includes: 

a.  Compilers 

b.  Assemblers 

c.  Link  editors 

d.  Loaders 

e.  Code  generators 

f.  Job  control  language  programs 

g.  Various  other  support  utilities 

These  needed  support  programs  must  be 
identified  in  engineering  documentation, 
usually  the  VDD,  to  each  specific  CPC  I 
version  and  the  materials  themselves 
maintained  in  the  CPL. 

6.2.5  Summary  of  Library  Controls 

The  CPL  function,  therefore  provides  for 
all  materials  needed  to  create  and  main¬ 
tain  programs  throughout  the  software 
development  life  cycle.  Usually  CPL  con¬ 
trols  will  have  been  established  early 
in  the  Full-Scale  Development  Phase  of 
the  prime  weapon  system  and  will  include 
all  weapon  system  software.  By  the  time 
the  ground  systems  are  ready  to  require 
CPL  controls,  the  existing  CPL  may 
already  contain: 

a.  Avionics  software 

b.  Special  Laboratory  Test  Software 

c.  Data  reduction  and  processing  pro¬ 
grams 

d.  Programmable  Read  Only  Memory 
(PROM)  software 
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e.  Avionics  simulation  and  support 
programs 

The  librarian's  job  therefore  becomes 
increasingly  more  complex  as  ATE/TS  pro¬ 
grams  are  added  to  his  facility.  Careful 
planning,  vigilant  maintenance  and  QA 
auditing  of  the  CPL  function  helps 
insure  success  of  the  software 
configuration  control  effort. 

6.3  PROBLEM  REPORTING  AND  CORRECTIVE 
ACTION 

Software  errors  are  a  diverse  and  diffi¬ 
cult  lot.  They  range  from  the  trivial 
(e.g.  syntactic  errors  resulting  from 
mis-use  of  a  language,  detected  by  the 
language  processor)  to  the  complex 
(e.g.,  of  #a  logic  error  resulting  only 
from  the  execution  of  a  peculiar  combina¬ 
tion  of  program  logic  paths,  detectable 
only  by  constructing  an  input  case  which 
forces  the  execution  of  that  combina¬ 
tion). 

Software  errors  emerge  from  all  phases 
of  the  software  life  cycle.  Recent  data 
tends  to  show  a  preponderance  of  errors 
resulting  from  the  design  phase.  This 
may  be  because  design  is  really  the  most 
error-prone  phase  of  the  life  cycle,  or 
it  may  be  because  most  error  reporting 
systems  do  not  begin  tracking  errors 
until  the  software  is  placed  under  con¬ 
figuration  control;  a  point  in  the  cycle 
when  most  coding  errors  have  been 
detected  and  corrected.  Be  that  as  it 
may,  the  source  of  errors  has  as  diverse 
and  difficult  a  picture  as  their  nature. 
The  detection  of  errors  results  from  a 
software  development  process  and  a  soft¬ 
ware  quality  process.  Errors  are  also 
detected  by  the  software  user.  The  soft¬ 
ware  quality  engineer,  for  example,  may 
detect  errors  through  reviews  and 
audits;  through  his  role  in  the  product 
test  function;  through  the  use  of  error¬ 
detecting  tools  whose  use  he  has  pro¬ 
moted  and  monitored;  and  through  the  for¬ 
mal  testing  process  he  helped  define  and 
execute.  Because  of  the  diversity  of 
errors,  their  sources,  and  their  ways  of 
detection,  a  central  control  of  error 
reporting  system  is  necessary.  The  SWQA 


engineers  should  either  motivate,  enable 
or  actually  operate  this  centrally- 
controlled  system. 

According  to  a  recent  Boeing  study,  (Bib¬ 
liography  ref.  10)  on  a  medium  sized  pro¬ 
ject,  the  primary  sources  of  errors  are 
logic,  such  as  missing  or  incorrect 
logic  (31.2%);  and  data  handling,  such 
as  the  mis-use  of  data,  indices  or  flags 
(13.4%).  No  other  single  source 
accounted  for  more  than  8%  of  errors 
encountered.  (Categories,  22  in  number 
were  provided  by  R ADC ) .  If  all  data- 
related  errors  were  totaled  (data 
handling,  data  base  interface  and  preset 
data  base  +  global  variable/compool 
definition),  the  result  is  19.8%.  Inter¬ 
estingly,  interface  errors  were  not  sig¬ 
nificant  (3.9%).  The  results  of  this 
study  are  shown  in  Figure  6.3-1.  While 
this  data  represents  that  of  an  airborne 
avionics  system  software  project,  the 
generic  causes  of  problems  are  uni¬ 
versally  applicable. 

6.3.1  Problem  Reporting  Methods  and 
Concepts 

It  is  important  to  report  and  track  soft¬ 
ware  problems.  It  is  also  important  to 
know  when  to  track,  and  for  how  long. 
Typically  in  the  software  life  cycle,  a 
great  amount  of  error  detection  will 
occur  before  a  formal  tracking  system  is 
necessary.  Errors  discovered  in  the  ana¬ 
lysis  and  design  phases,  for  example, 
are  generally  not  tracked  since  they  are 
usually  corrected  in  the  emerging  speci¬ 
fication  and  design  as  they  are  dis¬ 
covered.  Even  during  the  code  and  check¬ 
out  phase,  errors  usually  are  not 
tracked  because  they  are  too  numerous 
and  because  the  time  to  fix  is  often 
less  than  the  time  to  report.  It  is 
after  a  software  product  begins  accep¬ 
tance  testing,  or  is  released  to  the 
customer,  or  begins  system  integration 
(preferably,  whichever  comes  first), 
that  a  formal  tracking  system  becomes 
necessary. 

The  purpose  of  an  error  tracking  system 
is: 


X 


50 


w-j* !  ap 


.  ■■».'ll"  ..*$ 


CATEGORY 

number 

PERCENT  % 

A 

COMPUTATION 

100 

5.4 

B 

LOGIC 

635 

31.2 

C 

I/O 

28 

1.4 

D 

DATA  HANDLING 

272 

13.4 

E 

OS/SYS.  SUP.  SAV 

0 

0.4 

F 

CONFIGURATION 

12 

0.6 

G 

ROUTINE/ROUTINE  INTERFACE 

41 

2.0 

H 

ROUTINE/SYS.  S/W  INTERFACE 

3 

0.2 

l 

TAPE  PROCESSING  INTERFACE 

5 

0.3 

J 

USER  INTERFACE 

12 

0.6 

K 

DATABASE  INTERFACE 

17 

0.8 

L 

USER  REQUESTED  CHANGES 

161 

7.0 

M 

PRESET  DATA  BASE 

67 

3.3 

N 

GLOBAL  VARIABLE/CCMPOOL  DEF 

46 

2.3 

P 

RECURRENT 

148 

7.3 

Q 

DOCUMENTATION 

2? 

1.3 

R 

REQUIREMENTS  COMPLIANCE 

144 

7.1 

S 

UNIDENTIFIED 

30 

U 

T 

OPERATOR 

168 

7*3 

U 

QUESTIONS 

10 

0.0 

V 

HARDWARE 

32 

1.6 

X 

NON-REPRODUCIBLE 

62 

3.1 

Figure  6.3'  1 .  Software  Errors 
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a.  To  ensure  that  errors  are  cor¬ 
rected,  not  forgotten, 

b.  To  ensure  that  all  error  correc¬ 
tions  are  approved  before  changes  are 
made, 

c.  To  enable  the  measuring  of  error 
correction  progress, 

d.  To  provide  feedback  to  the  user  on 
error  status, 

e.  To  prioritize  the  order  in  which 
errors  are  corrected. 

The  role  of  SWQA,  in  this  discrepancy 
reporting  and  corrective  action  process, 
should  include  some  or  all  of  the 
following-definition  of  the  system, 
including  design  of  the  necessary  forms 
and  reporting  methodologies, 

a.  Tracking  open  error  reports,  to 
ensure  that  none  are  forgotten  and  that 
high  priority  problems  are  worked  first, 

b.  Participating  in  change  reviews, 
to  represent  the  quality  viewpoint  in 
the  decision-making  process. 

c.  Ensuring  that  error  status  is 
reported  to  the  software  users  (if  any), 

d.  Auditing  correction  progress  to  be 
sure  that  the  software  is  getting  better 
and  not  worse  (e,g,,  that  the  number  of 
uncorrected  errors  is  decreasing), 

6.3.2  Internal  Errors 

All  errors  reported  under  the  discrep¬ 
ancy  reporting  system  should  be  recorded 
on  a  Software  Problem  Report  (SPR)  form. 
The  SPR  becomes  the  primary  vehicle  for 
tracking  problems.  A  sample  SPR  form  is 
shown  in  Figure  6.3-2.  The  process  for 
processing  this  form  is  discussed  in  the 
succeeding  paragraphs  of  this  section. 

6.3.3  External  Errors 

If  an  error  is  detected  during  a  formal 
QA  controlled  test,  it  is  regarded  as 
official  test  discrepancy  (external  to 


the  software  development  organization), 
subject  to  formal  discrepancy  reporting 
corrective  action  mechanisms.  These  for¬ 
mal  mechanisms  are  costly  to  process 
since  they  usually  require  use  of  a  cor¬ 
porate  level  reporting  form;  copies  of 
which  are  sent  to  central  discrepancies 
accounting  organi zations.  These  types  of 
forms  usually  require  a  statement  of  cor¬ 
rective  action  prior  to  closure  since 
they  assume  the  discrepancy  being 
reported  is  a  true  deficiency  which 
should  not  have  occurred.  Daily,  routine 
software  development  errors  which,  from 
a  practical  standpoint  are  unavoidable, 
should  theoretical ly  not  occur  during  a 
formal  acceptance  test,  and  hence  not  be 
reported  on  corporate  discrepancy  track¬ 
ing  paperwork.  Care  must  therefore  be 
taken  in  early  stages  of  CPDP  prepara¬ 
tion,  not  to  submit  software  to  formal 
test  controls  prematurely. 

6.3.4  Problem  Report  Processing 

USAF  personnel  charged  with  acquisition 
or  subsequent  auditing  should  ensure 
that  the  contractor  implements  some  dis¬ 
ciplined  method  of  problem  report  pro¬ 
cessing.  Problem  report  processing 
shoul  d  entai 1 : 

a.  A  standard  serialized  reporting 
form 

b.  Analysis  for  impact 

c.  Categorization  for  trend  analysis 

(1)  self  contained-vs-interface 
error 

(2)  documentation  error 

(3)  enhancement  change 

(4)  test  case/procedure  error 

d.  Authorized  approval 

e.  Logging  and  monitoring  for  trends 

f.  Corrective  actions 

g.  Authorized  closure 

h.  Adequate  retest  and  engineering 
evaluation 
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Project  Name 


SOFTWARE  PROBLEM  REPORT 
Conputer/Lab  Utilized. 


Problem  Report  No. 
Program  _ 


Problem  Discovery 

Eethcd  or  detection: 

□  Usage 

□  Inspect  ion/Ana  lysis 


Name  of  finder 

□  Development  test 

□  Integration  test 
□Acceptance  Test 


Date 


Description  of  symptoms 


Tools  used  to  detect; 

□  None  □  Dump  (terminal)  □  Simulation 

□  Design  review  DDump  (dynamic)  ^Assertion 

□  Peer  review  qhDL  Debug  □  Proof 

D  Analyzer  □  Other 


Configuration  level 


Correction  importance/need  date 
Authorizing  Signature  _ 


Drgn. 


Date 


Problem  Analysis  Name  cf  analyst 
Findings 


Start  date  End  date 


Resources  expended:  person 


computer 


Estimated  resources  to  correct:  person 


computer 


Problem  Correction  Name  of  programmer 
Description  of  Correction  _ 


Start  date 


End  date 


Components  changed  and  conf iguration  level  _ 

Resources  expended:  Person  _ computer _ 

Problem  category  - 

□  Job  control  language  aDesign  error  -  emitted  logic 

□  Operational  Interface  DDesicn  error  -  faulty  logic 

□  Coding  error  -  data  declaration  OTesting 

□  Coding  error  -  executable  instruction  oCcnf iguration  management 


Final  Authorizing  Signature 


JDrgn._ 


□  Documentation 

□  Other 


Date 


Figure  6.3-2.  Suggested  Software  Problem  Report 
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i.  Adequate  engineering  change  cover¬ 
age 

6.3.5  Problem  Reporting  Summary 

It  must  be  emphasized  that  although  all 
software  errors  are  undesirable,  those 
encountered  during  a  formal  test  are 
most  expensive.  Designers  frequently 
claim  that  there  is  a  point  of  dimin¬ 
ishing  returns  reached  in  developmental 
testing;  that  no  matter  how  much  infor¬ 
mal  testing  is  done,  additional  errors 
will  inevitably  arise  from  exercising 
the  software  in  a  system  environment. 
This  is  certainly  true,  and  since  sched¬ 
ules  cannot  accomodate  a  so-called  “dry 
run"  (non-QC  witnessed)  of  a  entire  sys¬ 
tem  level  validation  test,  some  system 
level  software  problems  will  have  to  be 
handled  formally. 

6.4  VERIFICATION  AND  VALIDATION 
TES11NG  CONTROLS 


a.  The  article  under  test  n>j£t  !_*e  for 
rally  configured. 

b.  The  test  reqi.iren.ent:  **jst  be 
defined  in  released  engineering. 

c.  The  test  procedure  rrjf.c  be 
released  engineering. 

d.  Test  supoo't  equij-.nent  must  be  for¬ 
mally  configured,  accept  mico  tested  or 
cal ibrateJ/ct.'li^ied  as  i-.  pjired. 

e.  A  formal  test  planning  order  is 
required  (ref.  Fig.  6.1-1)  . 

f.  All  test  prerequisites  are  defined 
and  completed. 

g.  QA  witnessing  or  monitoring  of 
test  conduct  is  required. 

h.  Test  discrepancies  are  formally 
documented. 


In  paragraph  5.3  we  explored  the 
planning  aspects  for  verifying  that  soft¬ 
ware  meets  its  requirements.  We  identi¬ 
fied  the  ways  in  which  requirements  com¬ 
pliance  can  be  demonstrated,  what  to 
consider  in  planning  software  tests  and 
what  to  look  for  in  reviewing  test  plans 
and  procedures.  In  the  following  para¬ 
graphs,  we  shall  discuss  the  controls 
which  must  be  exercised  by  QA  in 
actually  conducting  or  monitoring  the 
conduct  of  formal  tests. 

6.4.1  Attributes  of  a  Formal  Test 

As  discussed  earlier,  the  contractor  per¬ 
forms  several  levels  of  testing  in  the 
development  of  CPCI's.  Module  verifica¬ 
tion  tests  and  module  compatibility 
tests  are  usually  regarded  as  engineer¬ 
ing  tests  or  confidence  builders  that 
the  design  is  proceeding  correctly.  They 
do  not  sell  off  or  acceptance  test  any¬ 
thing.  Formal  tests  on  the  other  hand, 
are  any  tests  designed  specifically  to 
provide  objective  evidence  of  satis¬ 
faction  of  released  engineering  require¬ 
ments.  Formal  tests  must  be  conducted 
under  the  auspices  of  QA  and  are 
characterized  by  the  following  controls. 


i.  Test  procedure  changes  require 
engineering  approval. 

j.  Formal  test  reports  are  prepared. 

k.  UUT  or  equipment  configuration 

changes  are  formally  released. 

l.  Test  set-up  must  be  formally 

def i ned. 


m.  Completed  records  contain  Mas  run" 
version  of  test  procedure. 

n.  Test  planning  order  contains  a  com¬ 
plete  history  of  test  events,  planned 
and  unplanned. 


o.  Formal  reviews  are  held  prior  to 
and  after  test. 

p.  QA  stamps  and  signs  the  UUT  and 
test  planning  order. 


Software  tests  designed  to 
compliance  with  CPCI  Part  I 
lent  documentation,  are 
formal  tests  requiring 
equivalent  to  the  above.  The 
conducting  formal  tests  is  as 


demonstrate 
or  equiva- 
considered 
controls 
sequence  of 
fol lows: 
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Release  Test  Plan 
Test  Readiness  Review  (TPR) 
Pre-Test  Briefing  (PTB) 
Test  Conduct 
Issue  Test  Report 


USAF 

►  Review 
and  Support 


QA  should  participate  in  all  of  the 
above  and  must  play  a  major  role  in  test 
conduct.  The  following  paragraphs  dis¬ 
cuss  the  main  events  involved  in  formal 
testing. 


6.4.2  Test  Readiness  Review  (TRR) 

The  TRR  is  a  formal  event  held  approxi¬ 
mately  two  weeks  prior  to  test  start 
with  USAF  in  attendance.  All  functional 
organizations  are  required  to  attend  to 
review  the  test  planning  status,  ident¬ 
ify  open  items  and  assign  action  items. 
A  typical  TRR  agenda  includes. 


a.  Statement  of  Test  Objectives, 
Requirements 


design;  who  are  authorized  to  commit  for 
their  organizations.  When  all  are 
satisfied  that  their  concerns  and  open 
items  are  identified  and  action  items 
assigned,  the  meeting  is  adjourned.  The 
chairman  follows  up  with  minutes  docu¬ 
menting  the  action  items. 

6.4.3  Pre-Test  Briefing  (PTB) 

The  PTB  is  held  to  assure  that  open 
items  identified  at  the  TRR  have  been 
acceptably  resolved  and  to  declare  "all 
is  go"  for  the  start  of  tests.  Some  of 
the  QA  concerns  to  be  registered  at 
these  reviews  should  be: 

a.  Is  the  specified  CPCI  version  to 
be  tested,  formally  configured  and 
released? 

b.  Are  any  machine  language  "hand¬ 
loads  or  patches"  required,  configured 
and  released? 


b.  Statement  of  Success  Criteria 

c.  Identification  of  Equipment,  Test 
Conductors,  Procedures 

d.  Status  of  Open  Items  Prerequi¬ 
sites 

e.  Description  of  Test 

f.  Facilities,  schedules 

g.  Documentation  Releases  -  Drawings, 
Planning  Orders,  etc. 

h.  Assignment  of  Action  Items  -  due 
dates 

The  TRR  insures  all  concerned  organiza¬ 
tions  have  the  opportunity  to  voice 
objections,  provide  corrections  and 
obtain  a  clear  understanding  of  the  test 
as  it  is  planned.  These  reviews  are  abso¬ 
lutely  essential  for  a  complex  test  pro¬ 
gram.  The  TRR  is  usually  chaired  by  the 
designated  lead  test  conductor  and  is 
attended  by  levels  of  management  from 
software  design,  systems  engineering,  QA 
manufacturing,  SWQA  and  equipment 


c.  Are  all  known  test  procedures 
discrepancies  resolved? 

d.  Is  all  required  test  support  equip¬ 
ment  properly  calibrated  or  certified? 

e.  Are  sufficient  QA  personnel  avail¬ 
able  to  support  planned  schedules? 

f.  Is  traceability  established  from 
the  CPCI  test  requirements  to  the 
procedures? 

Is  the  test  planning  order 
released  and  approved? 

h.  Is  all  required  test  support 
equipment  authorized? 

i.  Has  the  test  line  configuration 
been  verified? 

j.  Has  the  customer  been  notified  of 
test  schedules? 

k.  Is  the  particular  test  being 
planned  compatible  with  the  master 
software  verification  plan? 
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Most  contractor  project  QA  organizations 
d raw  support  from  central  QA  organiza¬ 
tions  -  factory  or  line  QA,  metrology 
services,  QA  engineering,  etc.  Whatever 
the  organization,  QA  should  be  ade¬ 
quately  represented  at  these  reviews. 
Line  QA  and  the  project  SWQA  engineer 
should  support  these  reviews  to  ade¬ 
quately  address  all  of  the  above 
concerns. 

6.4.4  Test  Conduct 

Usually  the  actual  "sell-off"  run  of  the 
test  will  be  preceded  by  a  so-called 
"dry  run."  This  dr7  run  is  conducted  by 
the  test  conductor  to  insure  its  sell- 
off  run  will  be  smooth,  clean  and  free 
of  discrepancies  requiring  costly  but 
necessary  discrepancy  handling  mecha¬ 
nisms.  It  also  eliminates  the  psycho¬ 
logical  hazzard  of  conducting  a  formal 
test  for  QA  and  the  customer,  which  is 
plagued  by  numerous  failures.  Unfortu¬ 
nately,  in  testing  complex  software  sys¬ 
tems,  schedules  oftentimes  do  not  permit 
dry  run  testing.  In  ATE  and  TS  software 
testing,  0A  should  expec*  software 
discrepancies  to  occur,  frequently 
requiring  changes  and  appropriate 
retest.  Therefore,  QA  must  be  extremely 
vigilant  during  the  monitorship  of  test¬ 
ing.  Unless  discrepancies  are  meticu¬ 
lously  documented,  software  configura¬ 
tion  control  can  easily  be  lost. 

The  first  question  which  must  be 
answered  is  -  should  QA  witness  100% 
every  step  of  the  procedure  or  will  a 
periodic  monitorship  with  spot-check 
witnessing  suffice?  This  can  only  be 
answered  by  evaluating  the  particular 
situation  at  hand  - 

a.  How  experienced  is  the  test 
conductor? 

b.  How  error-prone  is  the  procedure? 

c.  Are  numerous  recordings,  observa¬ 
tions  required? 

d.  Is  the  procedure  laborious  and 
tiresome? 


e.  Is  test  support  equipment  subject 
to  frequenty  failure? 

f.  Is  the  CPCI  subject  to  frequenty 
failure? 

Whatever  mode  of  monitorship  is  selected 
by  QA,  it  should  be  justified  and  speci¬ 
fically  called  out  by  the  test  planning 
order  -  "100%  witness"  or  "verify  comple¬ 
tion  of  — ."  Next,  QA  must  verify  test 
set-up.  Is  the  test  configuration 
assembled,  equipment  calibrated,  cables 
properly  hooked  up,  etc? 

The  next  operation  will  probably  call 
for  loading  the  CPCI  program  into  the 
memory.  This  operation  should  be 
witnessed  to  insure  the  authorized  soft¬ 
ware  plus  any  applicable  handlcads  have 
been  successfully  loaded.  A  common 
technique  for  verifying  successful  load 
is  the  "checksum  verification";  an  auto 
matic  comparison  of  a  bit  count  of 
loaded  data  with  a  pre-calcul ated  and 
stored  checksum.  This  is  usually 
required  as  part  of  the  procedure  and  is 
"bought  off"  by  the  inspector,  unce  the 
baseline  program  is  loaded,  testing 
begins  and  inspection  authenticates  that 
the  procedure  was  run  as  written.  In 
software  testing,  frequent  actual  and 
apparent  failures  are  encountered.  All 
such  occurrences  are  to  be  documented  in 
the  procedure  at  the  discrepant  step. 
Anomalies  fall  into  four  or  five  basic 
categories: 

a.  Hardware  failure 

b.  Software  failure 

c.  Procedural  discrepancy 

d.  Operator  error 

e.  Peripheral  interference 

Oftentimes,  in  software  testing,  peri¬ 
pheral  interference  can  cause  aggrava¬ 
ting  delays  while,  in  fact  nothing  is 
discrepant.  Radiated  or  conducted  peri¬ 
pheral  line  transients  (spikes  on  power 
lines,  on-off  cycling  of  heater 
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switches,  static  discharges,  etc.)  have 
been  proven  to  cause  apparent  test  fail¬ 
ures.  These  are  minimized  by  properly 
designing  the  test  environment  -  using 
line  filters,  isolation  transformers, 
screened  rooms,  etc.  QA,  however,  must 
assume  the  worst,  until  it  is  adequately 
proven  to  be  such  interference.  Upon 
encountering  any  discrepant  condition 
during  the  test,  the  first  step  should 
be  to  record  the  event,  even  if  it  is 
uncertain  as  to  cause.  Usually  the  con¬ 
tractor's  normal  test  conduct  proce¬ 
dures,  QA  manuals  etc.,  define  how  to 
process  test  discrepancies.  Usually  they 
are  adequate  for  handling  software  test 
discrepancies  as  well  as  hardware,  how¬ 
ever,  certain  questions  unique  to  soft¬ 
ware  should  be  considered: 

a.  At  what  point  in  the  test  proce¬ 
dure  was  the  problem  encountered? 

b.  What  was  the  fix?  -  machine  lan¬ 
guage  patch-or-newly  recompiled  program? 

c.  What  locations  and  data  were  modi¬ 
fied?  Where  is  this  information  docu¬ 
mented? 

d.  What  controls  insure  only  auto- 
rized  memory  locations  were  modified  to 
fix  a  software  problem  after  initial 
load? 

e.  What  retest  is  required? 

(1)  for  the  portion  of  program 

failing? 

(2)  for  possible  regression  of  the 

program  in  other  areas/ 

f.  At  what  point  in  the  procedure  was 
official  testing  resumed? 

g.  Who  authorized  the  fix  and  test 
restart? 

h.  Has  all  impacted  released  docu¬ 
mentation  been  changed?  -  (Part  II  spec, 
VDD,  Config.  Index,  etc.) 


The  above  questions  present  an  idea  of 
the  main  concerns  which  must  be  exer¬ 
cised  by  QA  personnel  responsible  for 
monitoring  test  conduct.  Each  test  sit¬ 
uation  must  be  individually  evaluated. 
To  be  effective,  QA  people  should  be 
vigilant  and  inquisitive  at  all  times. 
They  should  ultimately  be  able  to  show, 
from  the  completed  test  planning  order 
records,  exactly  what  configuration  of 
operational  software  resided  in  memory 
during  each  test  segment,  what  sources 
of  test  data  (real  time  or  batch)  were 
used  during  the  test,  what  anomalies 
were  encountered,  how  were  they 
resolved,  and  what  the  final  product  con¬ 
figuration  is. 

6.4.5  Test  Reports 

The  final  QA  activity  relating  to  verifi¬ 
cation  and  validation  testing  is  the 
review  of  the  test  report.  The  software 
QA  engineer  should  review  the  test 
report  for  the  following: 

a.  Have  the  planned  objectives  been 
satisfied? 

b.  Does  the  report  discuss  the  degree 
of  requirement  satisfaction  achieved  by 
the  test  results? 

c.  Are  unsatisfied  objectives 
addressed  in  terms  of  a  reschedule  test 
committment? 

d.  Is  there  a  complete  description  of 
the  test  environment? 

(1)  personnel  conducting  and  wit¬ 
nessing  the  test 

(2)  output  generated 

(3)  input  data 

(4)  discrepancies  and  anomalies 
encountered 

(5)  equipment  configuration  used 
for  test 
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e.  Is  the  test  report  approved  by 
authors  of  the  test  requirements? 


6.4.6  Test  Control  Summary 

The  most  important  point  to  remember  in 
QA  control  of  testing  is  that  records 
must  result  showing  exactly  what  was 
tested  to  what  procedures  in  satisfac¬ 
tion  of  what  requirements.  Test  records, 
in  Conjunction  with  analysis  and  inspec¬ 
tion  records,  form  a  total  software  veri¬ 
fication  package.  QA  must  oversee  the 
preparation  and  correctness  of  this 
total  compliance  package  to  be  used  in 
support  of  formal  audits  as  discussed  in 
the  next  paragraph. 

6.5  CONFIGURATION  AUDltS 

Configuration  Audits  are  conducted  by 
customer  personnel  at  fixed  points 
during  the  system  acquisition  process. 
As  defined  by  the  CM  plan,  a  Functional 
Configuration  Audit  (FCA)  is  normally 
scheduled  sometime  around  the  completion 
of  CPCI  qualification.  A  Physical  Config¬ 
uration  Audit  (PCA)  is  normally  sched¬ 
uled  after  fabrication  and  acceptance  of 
the  first  article.  For  an  ATE  system, 
for  example,  the  contractor  will  require 
an  FCA/PCA  on  the  vendor  supplied  por¬ 
tion  of  the  ATE  system  (basic  processor, 
test  station  and  operating  system  soft¬ 
ware)  prior  to  delivery  to  him.  The  Air 
Force,  in  turn  will  require  and  FCA/PCA 
on  the  entire  ATE  system  -  (processor, 
station,  OS,  adapter  and  UUT  test  pro¬ 
gram).  The  requirements  for  configura¬ 
tion  audits  are  specified  in 
MIL-STD-1521A  ar.d  are  addressed  in  depth 
in  the  guidebook  on  Reviews  and  Audits. 
The  SWQA  plan  should  include  a 
requirement  for  retaining  records  needed 
to  support  formal  audits.  As  discussed 
previously,  the  primary  form  of 
objective  records  used  in  support  of 
FCA/PCA  will  be  test  records.  Depending 
upon  provisions  of  the  contract,  the  PCA 
or  first  article  delivery  will  determine 
the  product  baseline  against  which 
formal  Class  I  chanqes  must  be 
processed,  and  incorporation  thereof 
verified  by  QA.  QA  records  must  be 
available  to  support  the  objectives  as 
outlined  below: 


6.5.1  FCA  Provisions 

a.  Verification  that  CPCI  qualifi¬ 

cation  requirements  including  test  plan, 
design,  test  procedures  and  results  are 
properly  documented. 

b.  Verification  that  the  CPCI  has 

been  subjected  to  all  specified  tests 
which  verify  its  performance. 

c.  Verification  that  the  CPCI  per¬ 
forms  as  required  by  its  performance 
specification. 

d.  Verification  that  all  approved 
changes  have  been  incorporated  and 
successfully  tested. 

e.  Verification  that  test  documenta¬ 
tion  is  current  and  accurate  with 

respect  to  the  configuration  of  the 
CPCI. 

6.5.2  PCA  Provisions 

a.  All  media,  listings  and  documenta¬ 
tion  comprising  the  deliverable  CPCI  are 
properly  identified. 

b.  The  CPCI  design  conforms  to  appli¬ 
cable  documentation  and  programming 
standards. 

c.  The  end  item  subjected  to  PCA  Is 
the  same  as  that  subjected  to  FCA. 

d.  QA  and  CM  records  adequately 
account  for  all  approved  changes.  The 
types  of  QA/CM  records  and  documents 
which  support  these  verification 
objectives  are: 

(1)  Completed  test  planning  orders 

(2)  Planning  accountability 

records 

(3)  Configuration  summaries 

(4)  Discrepancy  reports  (Unplanned 

Events) 

(5)  Configuration  Index 

(DI-E-3122) 


(6)  Change  Status  Lists 
(DI-E-3123) 

(7)  Version  Description  Document 
(DI-E-3121) 

(8)  Test  Requirements  Document 

(9)  CPCI  Part  I  and  II  specifica¬ 
tions 

(10)  Programming  standards  and  con¬ 
ventions 

(11)  Test  Reports  DI-T-3717 

(12)  User's  Manual  DI-3410 

(13)  Test  Plans/Procedures 
DI-T-3703 

6.5.3  FCA/PCA  Considerations 

In  order  to  prepare  for  a  successful  FCA 
and  PCA,  it  is  important  that  CPCI  docu¬ 
mentation  updates  keep  pace  with  the  pro¬ 


gressing  program  code.  Machine  language 
patches  or  recompilations  of  new  soiree 
code  to  produce  corrected  or  more  effi¬ 
cient  code,  should  be  accompanied  by 
corresponding  updates  to  the  CPCI  pro¬ 
duct  spec,  VDD,  ICD,  TRD  and  other  docu¬ 
mentation  impacted  by  program  configura¬ 
tion  changes.  Unless  this  is  rigorously 
enforced,  design  documentation  changes 
are  usually  put  off  for  update  "when 
time  permits."  This  results  in  program 
code  incompatibilities  with  the  design 
logic  narrative  and  flow  when  audited  at 
PCA.  Adequate  preparation  for  FCA/PCA, 
therefore  begins  at  CDR,  when  design 
begins  its  translation  in  code.  A 
successful  PCA  signals  the  completion  of 
the  software  development  process.  The 
CPCI  is  accepted  by  the  customer  and 
deployed  as  an  operational  component  of 
Operational  Support  Equipment,  (OSE). 
All  applicable  documentation  is  now 
under  Class  I  control.  The  utility  pro¬ 
vided  to  the  customer  by  the  CPCI  will 
ip  a  large  measure  be  a  reflection  of 
the  effectiveness  of  the  SWQA  program. 
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Section  8.0  MATRIX:  GUIDEBOOK  TOPICS  VS  GOVERNMENT  DOCUMENTS 


Figure  8.0-1  is  a  cross  reference  matrix 
showing  the  guidebook  topics  and  govern¬ 
ment  documents  which  address  each  topic. 


Elements  in  the  matrix  indicate  guide¬ 
book  sections  in  which  the  topic  is 
discussed. 
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Section  9.0  GLOSSARY  OF  TERMS 


Allocated  Basel ine  -  The  approved 
conf i gu rat ion  TtenT  identification.  It 
governs  the  development  of  selected 
configuration  items  that  are  part  of  a 
higher  level  specification,  e.g.,  system 
specification.  It  is  usually  defined  by 
the  Computer  Program  Development 
Specification. 

Basel ine  -  An  authorized  documented 
technical  description  specifying  an  end 
item's  functional  and  physical 
characteristics.  It  serves  as  the  basis 
for  configuration  control  and  status 
accounting.  It  establishes  an  approved 
well-defined  point  of  departure  for 
control  of  future  changes  to  system  or 
equipment. 

Certification  -  The  test  and  evaluation 
ot  the  complete  computer  program  aimed 
at  ensuring  operational  effectiveness 
and  suitability  with  respect  to  mission 
requirements  under  operating  conditions. 

Computer  Program  -  A  series  of 
instructions  or  statements  in  a  form 
acceptable  to  computer  equipment, 
designed  to  cause  the  execution  of  an 
operation  or  series  of  operations. 
Computer  programs  include  such  items  as 
operating  systems,  utility  programs,  and 
ma  i nt ena  nce/d i agnost i c  programs .  They 
also  include  applications  programs  such 
as  payroll,  inventory,  control,  opera¬ 
tional  flight,  strategic,  tactical,  auto¬ 
matic  test,  crew  simulator  and  engi¬ 
neering  analysis  programs.  Computer 
programs  nay  be  either  machine  dependent 
or  machine  independent,  and  may  be 
general  purpose  in  nature  or  be  designed 
to  satisfy  the  requirements  of  a  spec¬ 
ialized  process  of  a  particular  user. 

Computer  Program  Development  Cycle  -  The 
computer  program  development  cycle  con¬ 
sists  of  six  phases:  analysis,  design, 
coding  and  checkout,  test  and 
integration,  i nstal lat ion,  and  operation 
and  support.  The  cycle  may  span  more 
than  one  system  acquisition  life  cycle 
phase  or  may  be  contained  in  any  one 
phase.  (APR  800-14,  Volume  II) 


Computer  Program  Configuration  Items  -  A 
computer  program  or  aggregate  of  rel ated 
computer  programs  designated  for  configu¬ 
ration  management.  A  CPCI  may  be  a 
punched  deck  of  cards,  paper  or  magnetic 
tape  or  other  media  containing  a 
sequence  of  instructions  and  data  in  a 
form  suitble  for  insertion  in  a  digital 
computer. 

Configuration  Item  -  An  aggregation 
which  satisfies  an  end  use  function  and 
is  designated  for  configuration  manage¬ 
ment. 

Configuration  Management  -  A  management 
discipline  applying  technical  and  admin¬ 
istrative  direction  and  surveillance  to: 

a.  Identify  and  document  the  func¬ 
tional  and  physical  characteri sties  of  a 
configuration  item; 

b.  Control  changes  to  those  charac¬ 
teristics;  and 

c.  Record  and  report  change  pro¬ 
cessing  and  implementation  status. 

Control  Software  -  Software  used  during 
execution  of  a  test  program  which  con¬ 
trols  the  nontesting  operations  of  the 
ATE.  This  software  is  used  to  execute  a 
test  procedure  but  does  not  contain  any 
of  the  stimuli  or  measurement  parameters 
used  in  testing  a  unit  under  test.  Where 
test  software  and  control  software  are 
combined  in  one  inseparable  program, 
that  program  will  be  treated  as  +  est 
software.  (AFLC  66-37) 

Data  Base  -  A  collection  of  program 
code,  tables,  constants,  interface 
elements  and  other  data  essential  to  the 
operation  of  a  computer  program  or  soft¬ 
ware  subsystem. 

External  Interface  -  Data  passed  between 
two  or  more  computer  programs  or  between 
a  computer  program  and  peripheral 
devices  external  to  the  computer  in 
which  the  program  resides.  The  data  may 
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be  in  the  form  of  an  interrupt  signal  or 
may  be  a  digital  data  stream  either 
output  from  the  computer  or  input  into 
the  computer  for  processing. 

Hierarchial  Decomposition  -  A  process 
which  successively  divides  some  entity 
(requirement,  computer  process,  data 
element,  etc.)  into  components  pro¬ 
gressively  exposing  more  detail.  Each 
decomposition  is  a  complete  restatement 
of  the  parent  element  (and  only  the 
parent  element).  The  collection  of  all 
elements  at  any  one  level  is  a 
functionally  complete  entity. 

HIPO  Diagrams  (Hierarchy  plus  Input 
Proces~s  Output)  -  A  graphic  design 
technique  used  to  show  functions  in 
terms  of  the  input  to  a  process,  the 
process  itself,  and  the  output  results 
from  that  process.  A  typical  HIPO 
package  contains  three  different  items; 
a  table  of  contents,  overview  diagrams, 
and  detail  diagrams. 

Internal  Interfaces  -  Data  passed 
between  elements  of  a  computer  program 
and  usually  included  in  the  computer 
program  data  base. 

Logic  Flow  -  A  diagrammatic  representa¬ 
tion  of  the  logic  sequence  for  a  com¬ 
puter  program.  Logic  flows  may  take  the 
form  of  the  traditional  flow  charts  or 
in  some  other  form  such  as  a  program 
design  language. 

Naming  Convention  -  Standard  conventions 
applied  to  the  naming  of  variables,  data 
base  items,  modules  and  lower  level 
computer  program  components.  ^  Such 
conventions  should  allow  distinguishing 
a  computer  program  components  place  in 
the  hierarchial  structure,  by  its  common 
data  base  names  from  local  variable 
names,  and  common  subroutines  from 
module  unique  subroutines. 

Organic  -  A  term  used  to  designate  a 
t a sF  performed  by  the  Air  Force  rather 
than  a  contractor. 


Product  Baseline  -  The  final  approved 
conf l gurati  on  identification.  It 

identifies  the  as  designed  and  func¬ 
tionally  tested  computer  program  configu¬ 
ration.  It  is  defined  by  the  Computer 
Program  Product  Specification. 

Program  Design  Language  -  An  English- 
1  ike,  specially  formatted,  textual  lan¬ 
guage  describing  the  control  structure, 
logic  structure,  and  general  organi¬ 
zation  of  a  computer  program.  Essential 
features  of  a  program  design  language 
are: 

a.  It  is  an  English-like  represen¬ 
tation  of  a  computer  procedure  that  is 
easy  to  read  and  comprehend. 

b.  It  is  structured  in  the  sense  that 
it  utilizes  the  structured  programming 
control  structures  and  indentation  to 
show  nested  logic. 

c.  It  uses  full  words  or  phrases 
rather  than  the  graphic  symbols  used  in 
flow  charts  and  decision  tables. 

Software  -  A  combination  of  associated 
computer  programs  and  computer  data 
required  to  enable  the  computer  equip¬ 
ment  to  perform  computational  or  control 
functions. 

Software  Quality  -  The  primary 
characteristic  of  software  quality  is 
that  the  software  performs  as  intended. 
This  implies  not  only  that  the  software 
reflects  the  specification  to  which  it 
is  written,  but  also  that  the  software 
specifications  themselves  adequately 
address  the  system/mission  requirements. 
Key  attributes  of  software  quality  in¬ 
clude:  reliability,  flexibility,  trace- 
ability,  testability,  integrity,  main¬ 
tainability,  and  completeness.  Quality 
software  is:  well-defined,  well- 

documented,  free  of  design  deficiencies 
and  coding  errors,  satisfies  performance 
requirements,  and  has  minimum  life  cycl^ 
cost. 
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Software  Quality  Assurance  -  A  planned 
aTra  systematic  '"pattern  oF  all  software 
related  actions  necessary  to  provide 
adequate  confidence  that  Computer 
Program  Configuration  Items  (CPC I' s )  or 
other  software  products  conform  to 
established  technical  requirements  and 
that  they  achieve  satisfactory  results. 

Support  Software  -  Auxiliary  software 
used  to  aid  in  preparing,  analyzing  and 
maintaining  other  software.  Support 
software  is  never  used  during  the 

execution  of  a  test  program  on  a  tester, 

although  it  may  be  resident  either 
on-line  or  off-line.  Included  are 

assembl i es ,  compilers,  transl ators, 

loaders,  design  aids,  test  aids,  etc. 
(AFLC  66-37) 

System  Engineering  -  The  appl i cat i on  of 


scientific 

and 

engi neeri ng 

efforts 

to 

transform 

an 

operational 

need 

or 

statement 

of 

deficiency 

into 

a 

description 

of 

systems  reuqirements 

and 

a  preferred  system  configuration  that 
has  been  optimized  from  a  life  cycle 
viewpoint.  The  process  has  three 
principal  elements:  functional  analysis, 
synthesis,  and  trace  studies  or  cost- 
effectiveness  optimization. 

System  Generation  -  The  process  of 
producing  a  computer  program  from  two  or 
more  program  elements  such  that  the 
separate  program  elements  will  perform 
together  as  an  integrated  whole. 

System  Life  Cycle  -  The  system 
acquisition  life  cycle  consists  of  the 
following  five  major  phases  with  major 
decision  points: 

a.  Conceptual  phase 

b.  Validation  phase 

c.  Full-scale  development  phase 

d.  Production  phase 

e.  Deployment  phase 


Test  Software  -  Programs  which  implement 
documented  test  requirements.  There  is  a 
separate  test  program  written  for  each 
distinct  configuration  of  unit  under 
test.  (ALFC  66-37) 

Top  Down  Integration  -  The  development, 
testing  and  integration  of  the  separate 
structural  elements  of  a  computer 
program  in  a  hierarchial  manner 
beginning  with  the  highest  levels  and 
working  down  through  the  lowest  levels 
until  the  computer  program  is  complete. 

Top  Down  Structured  Programs 
Structured  programs  with  the  additional 
characteristics  of  the  source  code  being 
logically,  but  not  necessarily 
physically,  segmented  in  a  hierarchial 
manner  and  only  dependent  on  code 
already  written.  Control  of  execution 
between  segments  is  restricted  to 
transfer  between  vertically  adjacent 
hierarchial  segments. 

Val idat i on  -  Computer  program  validation 
is  the  test  and  evaluation  of  the 
complete  computer  program  aimed  at 
ensuring  compliance  with  the  performance 
and  design  criteria.  Validation  is 
generally  regarded  as  the  act  of  exer¬ 
cising  the  software  after  verification 
testing  in  a  real  or  simulated  operating 
environment  against  a  precisely  defined 
set  of  mission  or  test  case  requirements 
deemed  representative  of  the  usage  envir¬ 
onment  for  that  product.  Both  verifi¬ 
cation  and  validation  are  required  for 
software  acceptance. 

Verification  -  Computer  program 
verification  is  the  iterative  process  of 
continuously  determining  whether  the 
product  of  each  step  of  the  computer 
program  acquisition  process  fulfills  all 
requirements  levied  by  the  previous 
step,  including  those  set  for  quality. 
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Section  10.0  ABBREVIATIONS  AND  ACRONYMS 
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1 

ATE 

Automatic  Test  Equipment 

PDL 

Program  Design  Language 

( 

i 

ATLAS 

Abbreviated  Test  Language  for 

PDR 

Preliminary  Design  Review 

All  Systems 

PO 

Project  Office  (AF) 

! 

t 

i 

? 

y 

ATPG 

Automatic  Test  Program 

Generator 

PROM 

Programmable  Read  Only  Memory 

i 

• 

S* 

f' 

CCP 

Contract  Change  Proposal 

PTB 

Pre-test  Briefing 

t 

t 

» 

[ 

i 

CDR 

Critical  Design  Review 

QA 

Quality  Assurance 

CDRL 

Contract  Data  Requirements 

RFP 

Request  for  Proposal 

i  f 

V 

£* 

List 

ROM 

Read  Only  Memory 

1 

r 

,  r 

V 
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CEI 

Contract  End  Item 

SAE 

Software  Acquisition 

1 

*  • 

CM 

Configuration  Management 

Engineering 

I  < 

i 

t 

- 

CPCI 

Computer  Program  Configuration 
Item 

SCM 

Software  Configuration 
Management 

r 

< 

t 

CPDP 

Computer  Program  Development 

SDR 

System  Design  Review 

. 

* 

Plan 

SOW 

Statement  of  Work 

!» 

V 

$ 
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CPL 

Computer  Program  Library 

SPR 

Software  Problem  Report 

1 

| 

DID 

Data  Item  Description 

SRR 

System  Requirements  Review 

1  ‘  -] 

•  * 

V 
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DR 

Discrepancy  Report 

SWQA 

Software  Quality  Assurance 

|  j 

.  | 

.  I 

ECP 

Engineering  Change  Proposal 

TPR 

Test  Procedure  Review 

| 

- ! 

ERU 

Engineering  Release  Unit 

TRD 

Test  Requirements  Document 

f  > 

i 

i 

FCA 

Functional  Configuration  Audit 

TRR 

Test  Readiness  Review 

*• 

i 

HI  PO 

Hierarchi al -Input-Process- 
Output 

TS 

Training  Simulator 

t 

\ 

* 

HOL 

Higher  Order  Language 

UUT 

Unit  Under  Test 

i 

ICD 

Interface  Control  Drawing 

VDD 

Version  Description  Document 

i 

! 

5 

.  i 
•  < 

LRU 

Line  Replaceable  Unit 

r 

f 

i  ■ 

4 

v 

* 

I 

OS 

Operating  System 

L 

.  f 

OSE 

Operational  Support  Equipment 

1  - 

PAR 

Planning  Accountability  Record 

l 

PCA 

Physical  Configuration  Audit 
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